Training neural network model based on data point selection

ABSTRACT

An electronic device includes a memory to store neural network model trained for classification tasks of real-time applications. The neural network model is trained with plurality of training data points. The electronic device includes circuitry to retrieve a plurality of external data points. The electronic device applies the neural network model on the plurality of external data points to determine a plurality of impact scores for each external data point. The plurality of impact scores indicates amount of contribution of each training data point towards a prediction of each external data point. The electronic device selects a set of external data points based on the plurality of impact scores. The electronic device updates the plurality of training data points with the set of external data points to generate a second plurality of training data points and re-trains the neural network model based on the second plurality of training data points.

REFERENCE

None

FIELD

Various embodiments of the disclosure relate to training of a neuralnetwork model. More specifically, various embodiments of the disclosurerelate to an electronic device and a method for training of the neuralnetwork model based on external data point selection.

BACKGROUND

Recent advancements in the field of artificial intelligence have led todevelopment of various techniques of training artificial neural networkmodels (for example, a deep neural network (DNN) model and aconvolutional neural network (CNN) model). In certain situations, aneural network model may be iteratively trained to improve existingpredictive performance of the neural network model. However, thetraining of the neural network model may be a computationally expensiveand a time-consuming task. Moreover, usage of inappropriate externaldata to train or re-train the neural network model may deteriorate theexisting performance of the neural network model and further increasetraining time of the neural network model. Therefore, there exists aneed for a system which may control (i.e. minimize) the training timewhile maintaining high accuracy for the neural network model.

Further limitations and disadvantages of conventional and traditionalapproaches will become apparent to one of skill in the art, throughcomparison of described systems with some aspects of the presentdisclosure, as set forth in the remainder of the present application andwith reference to the drawings.

SUMMARY

An electronic device and method of training a neural network model basedon data point selection, are provided substantially as shown in, and/ordescribed in connection with, at least one of the figures, as set forthmore completely in the claims.

These and other features and advantages of the present disclosure may beappreciated from a review of the following detailed description of thepresent disclosure, along with the accompanying figures in which likereference numerals refer to like parts throughout.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates an exemplary networkenvironment for re-training of a first neural network model, inaccordance with an embodiment of the disclosure.

FIG. 2 is a block diagram that illustrates an exemplary electronicdevice of FIG. 1, in accordance with an embodiment of the disclosure.

FIG. 3 is a diagram that illustrates exemplary operations forre-training the first neural network model of FIG. 1 based on selectionof training data points, in accordance with an embodiment of thedisclosure.

FIG. 4 is a diagram that illustrates exemplary operations forre-training the first neural network model of FIG. 1 based on selectionof external data points, in accordance with an embodiment of thedisclosure.

FIG. 5 is a diagram that illustrates exemplary operations forre-training the first neural network model based on selection ofexternal data points with applied plurality of realistic variations, inaccordance with an embodiment of the disclosure.

FIG. 6 is a block diagram that illustrates exemplary dynamic tracking ofthe first neural network model, in accordance with an embodiment of thedisclosure.

FIGS. 7A-7B are block diagrams that illustrates exemplary operations ofthe electronic device to control a first neural network model and aplurality of second neural network models, in accordance with anembodiment of the disclosure.

FIG. 8 is a flowchart that illustrates an exemplary method forre-training of the first neural network model based on selection oftraining data points, in accordance with an embodiment of thedisclosure.

FIG. 9 is a flowchart that illustrates an exemplary method forre-training of the first neural network model based on selection ofexternal data points, in accordance with an embodiment of thedisclosure.

FIG. 10 is a flowchart that illustrates an exemplary method for dynamictracking of the first neural network model, in accordance with anembodiment of the disclosure.

DETAILED DESCRIPTION

The following described implementations may be found in the disclosedelectronic device and method to train a first neural network model.Exemplary aspects of the disclosure provide an electronic device thatmay include the first neural network model trained for a classificationtask of a real-time application. The first neural network model may betrained (for example, for prediction or classification of differentimages) with a first plurality of training data points (for example, butnot limited to, images of different breeds of dogs and certain images ofother objects, such as, birds or humans). However, all training datapoints of the first plurality of training data points may not be besttraining data points for the first neural network model. The electronicdevice may be configured to determine an impact score (for example, athird plurality of impact scores) for each of the first plurality oftraining data points. In some embodiments, the electronic device may beconfigured to update the first plurality of training data points byremoval of a set of training data points from the first plurality oftraining data points based on the determined impact score of each of thefirst plurality of training data points. The removal of the set oftraining data points may lead to data cleaning of the first neuralnetwork model for the set of training data points, for which thedetermined impact score may not meet a predefined selection criteria.

The electronic device may be configured to retrieve a first plurality ofexternal data points (for example, different images of different dogsand certain images of other objects, such as, birds or humans which maybe unknown to the trained first neural network model), which may bedifferent from the first plurality of training data points on which thefirst neural network model may be trained. The first plurality ofexternal data points may correspond to external data for the firstneural network model. The electronic device may further be configured toapply the first neural network model on the first plurality of externaldata points to determine a first plurality of impact scores for each ofthe first plurality of external data points. The first plurality ofimpact scores may indicate a first amount of contribution of each of thefirst plurality of training data points of the first neural networkmodel towards prediction of each of the first plurality of external datapoints.

In other words, the first amount of contribution may indicate aninfluence of each of the first plurality of training data points (i.e.images or other data types on which the first neural network model isalready trained) for the prediction or classification of each of thefirst plurality of external data points (i.e. images or other data typeswhich may correspond to external data which may be unknown to the firstneural network model). In accordance with an embodiment, the firstamount of contribution may relate to an amount of contribution offeatures of each of the first plurality of training data points that mayhave contributed towards the prediction of each of the first pluralityof external data points. For example, for the first neural network modeltrained to predict a dog from the first plurality of external datapoints (such as images), the features may include, but are not limitedto, a facial structure of the dog, a size of the dog, and a height ofthe dog. Each of the first plurality of impact scores may range from “0”to “1”.

The electronic device may be further configured to select a first set ofexternal data points from the first plurality of external data pointsbased on the determined first plurality of impact scores for each of thefirst plurality of external data points. The first plurality of impactscores of each of the selected first set of external data points may bemore than the first plurality of impact scores of remaining externaldata points in the first plurality of external data points. In anembodiment, the first plurality of impact scores of the selected firstset of external data points may be higher than an impact score thresholdor may be within a particular range of impact scores (such as Top N).Furthermore, the electronic device may be configured to update the firstplurality of training data points with the selected first set ofexternal data points to generate a second plurality of training datapoints (i.e. new training data points). The generated second pluralityof training data points may include the first plurality of training datapoints as well as the selected set of external data points. Theelectronic device may be further configured to re-train the first neuralnetwork model with the generated second plurality of training datapoints.

Thus, the electronic device of the present disclosure may enabledetermination of the first plurality of impact scores that may be usedin the selection of the first set of external data points from the firstplurality of external data points, for re-training of the first neuralnetwork model. The first plurality of external data points (i.e. unknownto the first neural network model) may thereby be prioritized orfiltered, to select the first set of external data points (i.e. withhigher impact score) for the re-training of the first neural networkmodel, instead of usage of all external data points in the firstplurality of external data points for the re-training. Further, theselected first set of external data points may be more effective for there-training of the first neural network model, as such external datapoints may be selected based on the determined impact scores for theexternal data points, instead of random selection. The first set ofexternal data points (i.e. say of higher impact scores) may thus beutilized for effective re-training of the first neural network model,which may improve an existing performance and accuracy of the firstneural network model. Further, the prioritization of the first pluralityof external data points and thereby the selection of the first set ofexternal data points may lead to a reduction of the number of externaldata points, thereby reducing a cost and time to re-train the firstneural network model. Therefore, the disclosed electronic device mayprovide cost effective as well as time efficient re-training of thefirst neural network model based on the selection of the first set ofexternal data points (i.e. external data).

In another embodiment, the electronic device may iteratively train thefirst neural network model (for example, as a dynamic tracking of thetraining of the first neural network model) for one or more epochs of aplurality of epochs (i.e. total number of predefined epochs) to generatea second neural network model. The second neural network model trainedon one or more epochs, may perform better prediction than the firstneural network model. The electronic device may determine an impactscore for each of the first plurality of training data points of thegenerated second neural network model trained for the one or moreepochs. The impact score may indicate an amount of contribution of eachof the first plurality of training data points of the generated secondneural network model towards a prediction of each of the first pluralityof training data points. The electronic device may re-select thegenerated second neural network model, as the first neural network modelfor training for others (i.e. remaining epochs) of the plurality ofepochs, based on a comparison between the determined impact score and atraining impact threshold. The electronic device may iteratively controlthe training of the first neural network model for each of the one ormore epochs of the remaining epochs, to obtain the second neural networkmodel (as a final trained neural network model) based on the comparison.In an example, when the determined impact score for each or a majorityof the first plurality of training data points exceeds the trainingimpact threshold, the electronic device may obtain the final neuralnetwork model (as the trained second neural network model) and stop theiterative training for the remaining epochs of the plurality of epochs.Thus, the dynamic tracking based on the impact determination, performedby the disclosed electronic device 102 may lead to a reduction in anumber of epochs required to train the first neural network model andthereby speed-up the training process to provide the final neuralnetwork model with expected accuracy.

FIG. 1 is a block diagram that illustrates an exemplary networkenvironment for re-training of a first neural network model, inaccordance with an embodiment of the disclosure. With reference to FIG.1, there is shown a network environment 100. The network environment 100may include an electronic device 102. The electronic device 102 mayfurther include a first neural network model 104 and a first pluralityof training data points 106. The first plurality of training data points106 may include a first training data point 106A, a second training datapoint 106B, . . . and an Nth training data point 106N. As shown in FIG.1, the network environment 100 may further include a database 108. Thedatabase 108 may include a first plurality of external data points 110.The first plurality of external data points 110 may include a firstexternal data point 110A, a second external data point 110B, . . . andan Nth external data point 110N. Furthermore, the network environment100 may include a communication network 112. The electronic device 102may be communicatively coupled to the database 108, via thecommunication network 112.

The electronic device 102 may include suitable logic, circuitry, code,and/or interfaces that may be configured to store the first neuralnetwork model 104 in a memory (shown in FIG. 2) of the electronic device102. Further, the electronic device 102 may determine a third pluralityof impact scores for the first plurality of training data points 106 anda first plurality of impact scores for each of the first plurality ofexternal data points 110. The electronic device 102 may be furtherconfigured to re-train the first neural network model 104 based on thedetermined third plurality of impact score for each of the firstplurality of training data points 106 and the first plurality of impactscores for each of the first plurality of external data points 110.Examples of the electronic device 102 may include, but are not limitedto, artificial intelligent (AI) machine, a computing device, asmartphone, a cellular phone, a mobile phone, a gaming device, amainframe machine, a server, a computer work-station, a tablet computer,a laptop computer, a desktop computer, and/or a consumer electronic (CE)device. In an embodiment, the first neural network model 104 may not bestored in the memory of the electronic device 102, but stored in thedatabase 108 communicably coupled to the electronic device 102, via thecommunication network 112.

The first neural network model 104 may be a computational network or asystem of artificial neurons, arranged in a plurality of layers, asnodes. The plurality of layers of the first neural network model 104 mayinclude an input layer, one or more hidden layers, and an output layer.Each layer of the plurality of layers may include one or more nodes (orartificial neurons, represented by circles, for example). Outputs of allnodes in the input layer may be coupled to at least one node of hiddenlayer(s). Similarly, inputs of each hidden layer may be coupled tooutputs of at least one node in other layers of the first neural networkmodel 104. Outputs of each hidden layer may be coupled to inputs of atleast one node in other layers of the first neural network model 104.Node(s) in the final layer may receive inputs from at least one hiddenlayer to output a result. The number of layers and the number of nodesin each layer may be determined from hyper-parameters of the firstneural network model 104. Such hyper-parameters may be set before orwhile training the first neural network model 104 on a training dataset,such as the first plurality of training data points 106.

Each node of the first neural network model 104 may correspond to amathematical function (e.g., a sigmoid function or a rectified linearunit) with a set of parameters, tunable while training of the firstneural network model 104. The set of parameters may include, forexample, a weight parameter, a regularization parameter, and the like.Each node may use the mathematical function to compute an output basedon one or more inputs from nodes in other layer(s) (e.g., previouslayer(s)) of the first neural network model 104. All or some of thenodes of the first neural network model 104 may correspond to same or adifferent mathematical function.

In training of the first neural network model 104, one or moreparameters of each node of the first neural network model 104 may beupdated based on whether an output of the final layer for a given input(from the training dataset) matches a correct result based on a lossfunction or not for the first neural network model 104. The aboveprocess may be repeated for the same or a different input till a minimaof loss function may be achieved and a training error may be minimized.Several methods for training are known in art, for example, gradientdescent, stochastic gradient descent, batch gradient descent, gradientboost, meta-heuristics, and the like.

The first neural network model 104 may include electronic data, such as,for example, a software program, code of the software program,libraries, applications, scripts, or other logic or instructions forexecution by a processing device, such as, circuitry of the electronicdevice 102. The first neural network model 104 may include code androutines configured to enable a computing device, such as, theelectronic device 102, to perform one or more operations forclassification of one or more inputs to one or more output labelsassociated with a real-time application. The first neural network model104 may be trained for a classification task of the real-timeapplication. Examples of the real-time application may include, but arenot limited to, an image recognition or classification application, aspeech recognition application, a text recognition application, amalware detection application, an autonomous vehicle application, ananomaly detection application, a machine translation application,pattern recognition from different digital signals, such as, but notlimited to, electrical bio signals, motion data, and depth data.Additionally, or alternatively, the first neural network model 104 maybe implemented using hardware including a processor, a microprocessor(e.g., to perform or control performance of one or more operations), afield-programmable gate array (FPGA), or an application-specificintegrated circuit (ASIC). Alternatively, in some embodiments, the firstneural network model 104 may be implemented using a combination ofhardware and software.

Examples of the first neural network model 104 may include, but are notlimited to, a deep neural network (DNN), a convolutional neural network(CNN), a recurrent neural network (RNN), a CNN-recurrent neural network(CNN-RNN), R-CNN, Fast R-CNN, Faster R-CNN, an artificial neural network(ANN), (You Only Look Once) YOLO network, a Long Short Term Memory(LSTM) network based RNN, CNN+ANN, LSTM+ANN, a gated recurrent unit(GRU)-based RNN, a fully connected neural network, a ConnectionistTemporal Classification (CTC) based RNN, a deep Bayesian neural network,a Generative Adversarial Network (GAN), and/or a combination of suchnetworks. In some embodiments, the first neural network model 104 mayinclude numerical computation techniques using data flow graphs. Incertain embodiments, the first neural network model 104 may be based ona hybrid architecture of multiple Deep Neural Networks.

In one or more embodiments, the first plurality of training data points106 may correspond to, but is not limited to, image data, audio data,text data or three-dimensional (3D) data. In an embodiment, the firstplurality of training data points 106 may correspond to electricalsignals. The first neural network model 104 may be trained with thefirst plurality of training data points 106. In an example, the firstneural network model 104 may be trained to predict an object in animage, such as an animal (e.g. a dog). In such as case, the firstplurality of training data points 106 may be images of the differenttypes of dogs. For example, the first training data point 106A may be animage of a dog of a first breed, the second training data point 106B maybe an image of the dog of a second breed, . . . and the Nth trainingdata point 106N may be an image of the dog of an Nth breed.

The database 108 may comprise suitable logic, circuitry, interfaces,and/or code that may be configured to store the first plurality ofexternal data points 110 related to the real-time application. Theelectronic device 102 may receive the first external data point 110A,the second external data point 110B, . . . and the Nth external datapoint 110N from the database 108. Further, the first plurality ofexternal data points 110 may be used to test or re-train the firstneural network model 104 for the real-time application. The firstplurality of external data points 110 may be unknown to the trainedfirst neural network model 104. Therefore, the first plurality ofexternal data points 110 may be referred as external data (or test data)on which the first neural network model 104 may not be trained. Thedatabase 108 may be a relational or a non-relational database that mayinclude the first plurality of external data points 110. Also, in somecases, the database 108 may be stored on a server, such as a cloudserver or may be cached and stored on the electronic device 102. Theserver of the database 108 may be configured to receive a request toprovide the first plurality of external data points 110 from theelectronic device 102, via the communication network 112. In response tosuch request, the server of the database 108 may be configured toretrieve and provide the first plurality of external data points 110 (orany of the first external data point 110A, the second external datapoint 110B, . . . or the Nth external data point 110N) to the electronicdevice 102, via the communication network 112. In some embodiments, thedatabase 108 may be configured to store the pre-trained first neuralnetwork model 104 for the particular real-time applications. In someembodiments, the database 108 may store the first plurality of impactscores for each of the first plurality of external data points 110 orthe third plurality of impact score for each of the first plurality oftraining data points 106. Additionally, or alternatively, the database108 may be implemented using hardware including a processor, amicroprocessor (e.g., to perform or control performance of one or moreoperations), a field-programmable gate array (FPGA), or anapplication-specific integrated circuit (ASIC). In some other instances,the database 108 may be implemented using a combination of hardware andsoftware.

The first plurality of external data points 110 may be utilized todetermine a predictability score of the first neural network model 104.The first plurality of external data points 110 may correspond to one ofthe image data, the audio data, the text data, or three-dimensional (3D)data. In an embodiment, the first plurality of external data points 110may correspond to electrical signals. For example, the first pluralityof external data points 110 may be provided to the first neural networkmodel 104 to determine the predictability score for the first pluralityof external data points 110. For example, the first external data point110A may be an image of a dog of a first breed, the second external datapoint 110B may be an image of a dog of a second breed, . . . and the Nthexternal data point 110N may be an image of a bird. The Nth externaldata point 110N (such as, an image that does not include a dog) may beprovided to the first neural network model 104 to ascertain that thefirst neural network model 104 may be able to determine an absence ofthe dog in the image or not (such as in the case of the Nth externaldata point 110N).

The communication network 112 may include a communication medium throughwhich the electronic device 102 and the database 108 may communicatewith each other. The communication network 112 may be one of a wiredconnection or a wireless connection. Examples of the communicationnetwork 112 may include, but are not limited to, the Internet, a cloudnetwork, a Wireless Fidelity (Wi-Fi) network, a Personal Area Network(PAN), a Local Area Network (LAN), or a Metropolitan Area Network (MAN).Various devices in the network environment 100 may be configured toconnect to the communication network 112 in accordance with variouswired and wireless communication protocols. Examples of such wired andwireless communication protocols may include, but are not limited to, atleast one of a Transmission Control Protocol and Internet Protocol(TCP/IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol(HTTP), File Transfer Protocol (FTP), Zig Bee, EDGE, IEEE 802.11, lightfidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hopcommunication, wireless access point (AP), device to devicecommunication, cellular communication protocols, and Bluetooth® (BT)communication protocols.

In operation, the electronic device 102 may receive a request tore-train the first neural network model 104 based on the first pluralityof external data points 110 (i.e. external data). The first neuralnetwork model 104 may be trained with the first plurality of trainingdata points 106. The electronic device 102 may be configured to select aset of training data points from the first plurality of training datapoints 106 based on a third plurality of impact scores determined foreach of the first plurality of training data points 106. The electronicdevice 102 may further update the first plurality of training datapoints 106 with removal of the selected set of training data points(i.e. from the first plurality of training data points 106), to furthergenerate a fourth plurality of training data points. The impact scorefor each of the selected set of training data points may not meet apredefined selection criteria. The fourth plurality of training datapoints may be utilized by the electronic device 102 to re-train thefirst neural network model 104. The details of the re-training of thefirst neural network model 104 with the fourth plurality of trainingdata points performed by the electronic device 102 is described, forexample, in FIG. 3. The removal of the selected set of training datapoints from the first plurality of training data points 106 may bereferred as data cleaning of the trained first neural network model 104based on the impact score determination.

Furthermore, the electronic device 102 may be configured to send arequest to the database 108 to retrieve the first plurality of externaldata points 110 from the database 108, via the communication network112. In some embodiments, the first plurality of external data points110 may be stored in the memory of the electronic device 102. Theelectronic device 102 may be further configured to apply the firstneural network model 104 on the first plurality of external data points110 to determine the first plurality of impact scores for each of thefirst plurality of external data points 110. The first plurality ofimpact scores may indicate the first amount of contribution of each ofthe first plurality of training data points 106 of the first neuralnetwork model 104 towards prediction of each of the first plurality ofexternal data points 110. In an example, the first neural network model104 may be trained for the prediction of the dog in an external image(such as the first external data point 110A) fed to the first neuralnetwork model 104. The first neural network model 104 may be trained onthe different images of dogs, such as dogs of different breeds andcertain images of other objects, such as, birds or humans. In such acase, the first plurality of training data points 106, such as the firsttraining data point 106A, the second training data point 106B, . . . andthe Nth training data point 106N may be the different images of the dogsand certain images of other objects, such as, birds or humans. Theelectronic device 102 may retrieve the first external data point 110Aand determine an impact (i.e. contribution or influence) of each of thefirst plurality of training data points 106 on the first external datapoint 110A (or towards the prediction or classification of the firstplurality of external data points 110 with a label “dog”). Thedetermined impact may correspond to an impact score for the firstexternal data point 110A. Similarly, the electronic device 102 maydetermine the first plurality of impact scores for each of the firstplurality of external data points 110. The determination of the firstplurality of impact scores for each of the first plurality of externaldata points 110 performed by the electronic device 102 is described, forexample, in FIG. 4.

The electronic device 102 may be further configured to select a firstset of data points from the first plurality of external data points 110based on the determined first plurality of impact scores. In anexemplary embodiment, an impact score for the first external data point110A may be “0.8”, an impact score for the second external data point110B may be “0.3”, . . . and an impact score for the Nth external datapoint 110N may be “0.7”. The electronic device 102 may select the firstexternal data point 110A and the Nth external data point 110N from thefirst plurality of external data points 110 as the first set of externaldata points. The electronic device 102 may select the first externaldata point 110A and the Nth external data point 110N, as the impactscore for the first external data point 110A and the impact score forthe Nth external data point 110N may be substantially more than theimpact score for the second external data point 1108 and/or more than animpact score threshold (e.g., “0.55” defined for the first neuralnetwork model 104 or for the classification task). The details of theselection of the first set of external data points from the firstplurality of external data points 110 performed by the electronic device102 is described, for example, in FIGS. 3 and 4.

The electronic device 102 may be further configured to update the firstplurality of training data points 106 with the selected first set ofexternal data points to generate a second plurality of training datapoints (not shown in FIG. 1). The second plurality of training datapoints may include the first plurality of training data points 106 andthe selected first set of external data points. The electronic device102 may further re-train the first neural network model 104 with thegenerated second plurality of training data points. The details of there-training of the first neural network model 104 performed by theelectronic device 102 is described, for example, in FIGS. 3 and 4. Asthe selection of the first set of external data points performed by theelectronic device 102 is based on the first plurality of impact scores,the selected first set of external data points may be more effective forthe re-training of the first neural network model 104, than theunselected external data points in the first plurality of external datapoints 110. Further, the selection of the first set of external datapoints from the first plurality of external data points 110 may reducethe number of data points that may correspond to best external datapoints used for re-training of the first neural network model 104. Thus,the re-training of the first neural network model 104 with the selectedfirst set of external data points may be efficient to provide a neuralnetwork model with high accuracy data prediction and classification.

In accordance with an embodiment, the electronic device 102 may befurther configured to apply a plurality of realistic variations (such asa plurality of augmentation techniques) to one or more external datapoints of the first plurality of external data points 110 to generate asecond plurality of external data points. The electronic device 102 mayfurther select a second set of external data points from the generatedsecond plurality of external data points based on a second plurality ofimpact scores determined for the second plurality of external datapoints. The electronic device 102 may generate a third plurality oftraining data points that may include the first plurality of trainingdata points 106 and the selected second set of external data points. Thethird plurality of training data points may be utilized to re-train thefirst neural network model 104. The details of the re-training of thefirst neural network model 104 with the third plurality of training datapoints performed by the electronic device 102 is described, for example,in FIG. 5.

In accordance with an embodiment, the electronic device 102 may befurther configured to dynamically track the training of the first neuralnetwork model 104. The electronic device 102 may be configured to selectthe first neural network model 104 for training for a plurality ofepochs (for example predefined number of epochs), while training thefirst neural network model 104 based on the first plurality of trainingdata points 106. The electronic device 102 may be further configured tocontrol a set of operations (i.e. dynamic tracking) for training theselected first neural network model 104 for one or more epochs out ofthe plurality of epochs based on impact score determination. The detailsof the dynamic tracking of the training of the first neural networkmodel 104 performed by the electronic device 102 is described, forexample, in FIG. 6.

In accordance with an embodiment, the electronic device 102 may befurther configured to dynamically select a neural network model from aplurality of second neural network models in production and stagingphase of the first neural network model 104. The electronic device 102may select the neural network model from the plurality of second neuralnetwork models based on a plurality of impact scores determined for asecond external data point with respect to the plurality of secondneural network models. In an embodiment, the electronic device 102 mayswitch the first neural network model 104 from the production phase withthe selected neural network model in the staging phase. The details ofthe selection of the neural network model from the plurality of secondneural network models, and switching of the selected neural networkmodel with the first neural network model 104 performed by theelectronic device 102, is described, for example, in FIG. 7A. Moreover,the electronic device 102 may add the selected neural network model inthe production phase along with the first neural network model 104. Thedetails of the addition of the selected neural network model in theproduction phase, is described, for example, in FIG. 7B.

FIG. 2 is a block diagram that illustrates an exemplary electronicdevice of FIG. 1, in accordance with an embodiment of the disclosure.FIG. 1 is described in conjunction with elements from FIG. 1. Withreference to FIG. 2, there is shown a block diagram 200 of electronicdevice 102. The electronic device 102 may include the first neuralnetwork model 104, such that the first neural network model 104 may betrained with the first plurality of training data points 106. Theelectronic device 102 may further include circuitry 202, a memory 204,an input/output (I/O) device 206, and a network interface 208, throughwhich the electronic device 102 may be connected to the communicationnetwork 112.

The circuitry 202 may include suitable logic, circuitry, and/orinterfaces that may be configured to execute program instructionsassociated with different operations to be executed by the electronicdevice 102. For example, some of the operations may include re-trainingthe first neural network model 104 based on the determination of thefirst plurality of impact scores and the third plurality of impactscores. The circuitry 202 may be further configured to dynamically trackthe training of the first neural network model 104. The circuitry 202may include one or more specialized processing units, which may beimplemented as a separate processor. In an embodiment, the one or morespecialized processing units may be implemented as an integratedprocessor or a cluster of processors that perform the functions of theone or more specialized processing units, collectively. The circuitry202 may be implemented based on a number of processor technologies knownin the art. Examples of implementations of the circuitry 202 may be anX86-based processor, a Graphics Processing Unit (GPU), a ReducedInstruction Set Computing (RISC) processor, an Application-SpecificIntegrated Circuit (ASIC) processor, a Complex Instruction Set Computing(CISC) processor, a microcontroller, a central processing unit (CPU),and/or other control circuits.

The memory 204 may include suitable logic, circuitry, and/or interfacesthat may be configured to store the one or more instructions to beexecuted by the circuitry 202. In accordance with an embodiment, thememory 204 may be configured to store the first neural network model 104trained for the classification task of the real-time application. Thememory 204 may be further configured to store the first plurality oftraining data points 106. In some embodiments, the memory 204 may beconfigured to store the first plurality of external data points 110. Inan embodiment, the memory 204 may store the first plurality of impactscores determined for the first plurality of external data points 110,and store the first impact score threshold which may be used to selectthe first set of external data points (such as the first external datapoint 110A and the second external data point 110B) from the firstplurality of external data points 110. Examples of implementation of thememory 204 may include, but are not limited to, Random Access Memory(RAM), Read Only Memory (ROM), Electrically Erasable ProgrammableRead-Only Memory (EEPROM), Hard Disk Drive (HDD), a Solid-State Drive(SSD), a CPU cache, and/or a Secure Digital (SD) card.

The I/O device 206 may include suitable logic, circuitry, code, and/orinterfaces that may be configured to receive an input from a user andprovide an output based on the received input. For example, theelectronic device 102 may receive the input to initiate the selection ofthe first set of external data points from the first plurality ofexternal data points 110, via the I/O device 206. In another example,the electronic device 102 may output the determined impact scores forthe selected first set of external data points, via I/O device 206. TheI/O device 206 which may include various input and output devices, maybe configured to communicate with the circuitry 202. Examples of the I/Odevice 206 may include, but are not limited to, a touch screen, akeyboard, a mouse, a joystick, a microphone, a display device, and aspeaker.

The network interface 208 may include suitable logic, circuitry, code,and/or interfaces that may be configured to facilitate communicationbetween the circuitry 202 and the database 108, via the communicationnetwork 112. The network interface 208 may be implemented by use ofvarious known technologies to support wired or wireless communication ofthe electronic device 102 with the communication network 112. Thenetwork interface 208 may include, but is not limited to, an antenna, aradio frequency (RF) transceiver, one or more amplifiers, a tuner, oneor more oscillators, a digital signal processor, a coder-decoder (CODEC)chipset, a subscriber identity module (SIM) card, or a local buffercircuitry. The network interface 208 may be configured to communicatevia wireless communication with networks, such as the Internet, anIntranet, or a wireless network, such as a cellular telephone network, awireless local area network (LAN), and a metropolitan area network(MAN). The wireless communication may be configured to use one or moreof a plurality of communication standards, protocols and technologies,such as Global System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), wideband code division multiple access (W-CDMA),Long Term Evolution (LTE), code division multiple access (CDMA), timedivision multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi)(such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g or IEEE 802.11n),voice over Internet Protocol (VoIP), light fidelity (Li-Fi), WorldwideInteroperability for Microwave Access (Wi-MAX), a protocol for email,instant messaging, and a Short Message Service (SMS).

A person of ordinary skill in the art will understand that theelectronic device 102 in FIG. 2 may also include other suitablecomponents or systems, in addition to the components or systems whichare illustrated herein to describe and explain the function andoperation of the present disclosure. A detailed description for theother components or systems of the electronic device 102 has beenomitted from the disclosure for the sake of brevity. The operations ofthe circuitry 202 are further described, for example, in FIGS. 1, 3, 4,5, 6, 7A, 7B, 8, and 9.

FIG. 3 is a diagram that illustrates exemplary operations forre-training the first neural network model of FIG.1 based on selectionof training data points, in accordance with an embodiment of thedisclosure. FIG. 3 is explained in conjunction with elements from FIGS.1 and 2. With reference to FIG. 3, there is shown a diagram 300 todepict exemplary operations from 302 to 310. The exemplary operationsillustrated in the diagram 300 may start at 302 and may be performed byany computing system, apparatus, or device, such as, by the electronicdevice 102 of FIG. 1 or the circuitry 202 of FIG. 2.

At 302, a third plurality of impact scores may be determined. Inaccordance with an embodiment, the circuitry 202 may be configured todetermine the third plurality of impact scores for each training datapoint of the first plurality of training data points 302A of a firstneural network model 302B (i.e. similar to the first neural networkmodel 104 in FIG.1). The first neural network model 302B may be trainedfor a classification task of a real-time application. In accordance withan embodiment, the real-time application may comprise an imageclassification, a speech recognition, or a text recognition task thatmay be performed by the first neural network model 302B. In accordancewith an embodiment, the first neural network model 302B may be trainedon the first plurality of training data points 302A (for example, dogimages) for recognition of different objects in a first plurality ofexternal data points (for example, test data points, not shown in FIG.3). In another example, the first neural network model 302B mayrecognize different audio samples in the first plurality of externaldata points to identify a source (e.g., a human-speaker) of the audiosample.

The circuitry 202 may determine the third plurality of impact scores foreach training data point of the first plurality of training data points302A by application of the first neural network model 302B on eachtraining data point of the first plurality of training data points 302A.The circuitry 202 may determine the third plurality of impact scores foreach training data point of the first plurality of training data points302A by comparison of features of each training data point withcorresponding features of all training data points in the firstplurality of training data points 302A. The determined third pluralityof impact scores may indicate a third amount of contribution (orinfluence) of each of the first plurality of training data points 302Aof the first neural network model 302B towards prediction of each of thefirst plurality of training data points 302A. In other words, determinedthird plurality of impact scores may indicate that how much eachtraining data point contributes or impacts in the prediction of anothertraining data point in the first plurality of training data points 302A.

In accordance with an embodiment, the circuitry 202 may be configured togenerate a first graphical representation 302C that may be indicative ofa relation between each training data point of the first plurality oftraining data points 302A. The first graphical representation 302C mayinclude a plurality of nodes and a plurality of edges between theplurality of nodes. Each node of the plurality of nodes may represent atraining data point of the first plurality of training data points 302A.Each edge of the plurality of edges may represent an impact score (or aweightage value) of the third plurality of impact scores for eachtraining data point. The impact score may indicate the third amount ofcontribution of each of the first plurality of training data points 302Aof the first neural network model 302B towards a prediction of thecorresponding training data point. As shown in FIG. 3, the firstgraphical representation 302C may include a first node for a trainingdata point “T1”, a second node for a training data point “T2”, a thirdnode for a training data point “T3”, a fourth node for a training datapoint “T4”, . . . and an Nth node as a training data point “Tn”. Theplurality of edges may indicate an impact score W1, an impact score W2,an impact score W3, an impact score W4, . . . and an impact score Wn, asthe third plurality of impact scores. In one or more embodiments, thethird plurality of impact scores, such as, the impact score W1, theimpact score W2, the impact score W3, the impact score W4, . . . and theimpact score Wn, may be any value between “0” and “1”. The generation ofthe first graphical representation 302C by the circuitry 202 may enablea determination of an impact of a first training data point of the firstplurality of training data points 302A on a second training data pointof the first plurality of training data points 302A, and vice versa.

In an embodiment, each edge between the first training data point andthe second training data point, associated with the first graphicalrepresentation 302C, may have an associated width that may be indicativeof an impact score for the first training data point towards theprediction of the second training data point. For example, the width ofan edge may be thicker to represent a higher impact score and thinner torepresent a lower impact score. In another embodiment, each edge of thefirst graphical representation 302C may have an associated color thatmay be indicative of an impact score of the first training data pointsfor the second training data point. For example, a darker color (suchas, black, brown, or red) of an edge may represent a higher impact scoreand a lighter color (such as, light blue, pale yellow, or cyan) of anedge may represent a lower impact score.

An exemplary relation between each training data point of the firstplurality of training data points 302A based on the third plurality ofimpact scores may be indicated by a matrix of the third plurality ofimpact scores, as depicted in Table 1:

TABLE 1 The third plurality of impact scores for each training datapoint of the first plurality of training data points 302A T1 T2 T3 T4 TnT1 0.5 0.1 0.3 0.34 0.11 T2 0.43 0.6 0.4 0.22 0.12 T3 0.12 0.3 0.8 0.320.26 T4 0.23 0.32 0.01 0.4 0.28 Tn 0.02 0.12 0.28 0.22 0.3

In accordance with an embodiment, the circuitry 202 may apply the firstneural network model 302B to the training data point “T1” to generatethe third plurality of impact scores for the training data point “T1”.For example, with reference to Table 1, an impact score of the trainingdata point “T1” towards itself (such as the training data point “T1”)may be “0.5”, an impact score of the training data point “T2” towardsthe training data point “T1” may be “0.43”, and an impact score of thetraining data point “T3” towards the training data point “T1” may be“0.12”. Further, an impact score of the training data point “T4” towardsthe training data point “Ti” may be “0.23”, and an impact score of thetraining data point “Tn” towards the training data point “T1” may be“0.02”. Therefore, the third plurality of impact scores for the trainingdata point “T1” are “0.5, 0.43, 0.12, 0.23, and 0.02” as per Table 1.Similarly, the impact scores for each training data point of the firstplurality of training data points 302A may be determined. As shown inthe Table 1, the impact score of a training data point towards itselfmay be maximum. It should be noted that data provided in Table 1 maymerely be taken as experimental data and may not be construed to limitthe present disclosure.

Though FIG. 3 is described for the first graphical representation 302Cas a relation between each training data point of the first plurality oftraining data points 302A, the scope of the disclosure may not be solimited. In some embodiments, alternatively or in addition to the firstgraphical representation 302C, the circuitry 202 may be configured togenerate a second graphical representation (not shown in FIG. 3) as arelation between each external data point and the first plurality oftraining data points 106, as described further for example, in FIG. 4.

In accordance with an embodiment, the circuitry 202 is furtherconfigured to apply a mathematical function to the determined thirdplurality of impact scores for each of the first plurality of trainingdata points 302A. The mathematical function may include, but is notlimited to, one of: a maxima function, an average function, a meanfunction, or a summation function. For example, based on the applicationof the average function, the circuitry 202 may determine an impact score(i.e. one score) for each of the first plurality of training data points302A. In an exemplary embodiment, the circuitry 202 may apply theaverage function (as the mathematical function) on the impact scores(i.e. “0.5, 0.43, 0.12, 0.23, and 0.02”) to determine the impact score“0.26” (i.e. average of “0.5, 0.43, 0.12, 0.23, and 0.02”) for thetraining data point “T1”. Similarly, the circuitry 202 may determine theimpact score for each training data point (“T1”, “T2”, “T3”, “T4”, andTn”) based on the application of the mathematical function on the thirdplurality of impact scores determined for each training data point asper Table 1). In another example, in case of the application of themaxima function on the third plurality of impact scores, the impactscore for the training data point “T1” is “0.5”.

In accordance with an embodiment, the circuitry 202 may be configured togenerate a prediction score for each of the first plurality of trainingdata points 302A. The prediction score may indicate a confidence levelof the first neural network model 302B in the prediction orclassification of an output class label (such as a dog class label) foreach of the first plurality of training data points 302A. In someembodiments, the circuitry 202 may be configured to apply the firstneural network model 302B on each of the first plurality of trainingdata points 302A to generate the prediction score for each of the firstplurality of training data points 302A. In an example, the first neuralnetwork model 302B may predict the training data point “T1” accuratelyas an image of the dog, and the prediction score for the training datapoint “Ti” may be determined as “0.9”. In another example, the firstneural network model 302B may predict another training data point of thefirst plurality of training data points 302A inaccurately as an image ofthe dog, and the prediction score for the training data point may bedetermined as “0.2”.

At 304, a set of training data points may be selected. In accordancewith an embodiment, the circuitry 202 may be configured to select theset of training data points from the first plurality of training datapoints 302A based on the determined third plurality of impact scores foreach of the first plurality of training data points 302A. The set oftraining data points may be a subset of the first plurality of trainingdata points 302A. In some embodiments, the circuitry 202 may beconfigured to select the set of training data points from the firstplurality of training data points 302A based on the determined thirdplurality of impact scores and the generated prediction score for eachof the first plurality of training data points 302A.

The selection of the set of training data points from the firstplurality of training data points 302A may be based on a predefinedselection criteria304A. In accordance with an embodiment, the circuitry202 may select the set of training data points based on the predefinedselection criteria 304A. In some embodiments, the circuitry 202 may beconfigured to select the set of training data points from the firstplurality of training data points 302A based on a first range of impactscores. In such a scenario, the first range of impact scores may be thepredefined selection criteria 304A. Thus, the set of training datapoints may include training data points that may have an impact scorethat may lie within the first range of impact scores. In an embodiment,the first range of impact scores may be between “0.1” and “0.5”. In sucha case, the training data point “T4” and the training data point “Tn”may be selected in the set of training data points, as the impact scoreof the training data point “T4” and the training data point “Tn” may be“0.4” and “0.3” respectively, based on the application of the maximafunction (which lies in the first range of impact scores). Similarly,other training data points in the first plurality of training datapoints 302A, whose impact score may lie in the first range of impactscores may be selected in the set of training data points.

In some embodiments, the circuitry 202 may be configured to select theset of training data points from the first plurality of training datapoints 302A based on the predefined selection criteria 304A, such as, asecond impact score threshold. The circuitry 202 may select the set oftraining data points such that each training data point in the selectedset of training data points may have the impact score less than thesecond impact score threshold. For example, the second impact scorethreshold may be “0.25”. The circuitry 202 may select the set oftraining data points of the first plurality of training data points 302Aas training data points with the impact scores (for example average)less than the second impact score threshold of “0.25”. In such ascenario, with reference to Table 1, the circuitry 202 may select thetraining data point Tn, as the impact score (i.e. average score) for thetraining data point “Tn” may be “0.21” (i.e. less than “0.25” as thesecond impact score threshold). Similarly, based on the application ofdifferent predefined mathematical function and the second impact scorethreshold, different training data points of the first plurality oftraining data points 302A may be selected in the set of training datapoints.

In some embodiments, the predefined selection criteria 304A for theselection of the set of training data points from the first plurality oftraining data points 302A may be, for example, “n” number of highestpositive impact scores (such as Top-N), or “n” number of highestnegative impact scores (such as Least-N) and so forth. For example, theimpact scores determined for the first plurality of training data points302A may be ordered (such an increasing or decreasing order), to selectthe set of training data points based on the predefined selectioncriteria 304A (such as Top-N or Least N). For example, as per Table 1,the training data point “T4” and the training data point “Tn” may beselected as the set of training data points based on the predefinedselection criteria 304A, such as Least-2. In an embodiment, thepredefined selection criteria 304A for the selection of the set oftraining data points may include a combination of the impact score andthe prediction score for each of the first plurality of training datapoints 302A. For example, the predefined selection criteria 304A mayinclude different scores thresholds (such as the second impact scorethreshold and a prediction threshold) to select the set of training datapoints from the first plurality of training data points 302A.

In accordance with an embodiment, the circuitry 202 may be configured togenerate a first matrix representation 312 to depict the first pluralityof training data points 302A based on the impact score (i.e. “IS”) andthe prediction score (“PS”) as shown in FIG. 3. The impact score may berepresented on an X-axis of the first matrix representation 312, whereasthe prediction score may be represented on a Y-axis of the first matrixrepresentation 312. The first matrix representation 312 may include afirst quadrant 312A, a second quadrant 312B, a third quadrant 312C and afourth quadrant 312D. In an exemplary embodiment, the training datapoints of the first plurality of training data points 302A that may havethe prediction scores between a range of “0.5” and “1.0” may beclassified as the training data points with a high prediction score.Further, the training data points of the first plurality of trainingdata points 302A that may have the prediction scores between a range of“0” and “0.5” may be classified as the training data points with a lowprediction score. Similarly, the training data points of the firstplurality of training data points 302A that may have the impact scorebetween a range of “0.5” and “1” may be classified as the training datapoints with a high impact score. Further, the training data points thatmay have the prediction scores between a range of “0” and “0.5” may beclassified as the training data points with a low impact score, as shownin the first matrix representation 312 in FIG. 3.

The training data points of the first plurality of training data points302A that lie in the first quadrant 312A may have the high predictionscore and the high impact score. The training data points of the firstplurality of training data points 302A that lie in the second quadrant312B may have the high prediction score and the low impact score. Thetraining data points of the first plurality of training data points 302Athat lie in the third quadrant 312C may have the low prediction scoreand the low impact score. The training data points of the firstplurality of training data points 302A that lie in the fourth quadrant312D may have the low prediction score and the high impact score, asshown in the first matrix representation 312 in FIG. 3. In accordancewith an embodiment, the circuitry 202 may select the training datapoints (i.e. from the first plurality of training data points 302A) thatmay lie in the first quadrant 312A and the fourth quadrant 312D, as theset of training data points.

In accordance with an embodiment, the circuitry 202 may be configured toselect a particular predefined selection criteria 304A (such as impactrange based, impact threshold based, order based, prediction based, orcombination) based on different factors, such as, but not limited to,the real-time application, a type of the first plurality of trainingdata points 302A, or a type of the first neural network model 302B.

At 306, the selected set of training data points may be removed. Inaccordance with an embodiment, the circuitry 202 may be configured toremove the selected set of training data points from the first pluralityof training data points 302A. The removal of the selected set oftraining data points (i.e. with lower impact scores) from the firstplurality of training data points 302A may ensure that unwanted trainingdata points may be cleansed from the first plurality of training datapoints 302A. The removal of such unwanted training data points mayimprove the performance of the first neural network model 302B, based onre-training of the first neural network model 302B on remaining of thefirst plurality of training data points 302A. This may be because theremoved set of training data points may have a lesser impact on aprediction output of the first neural network model 302B and may insteadlead to introduction of an overfitting or bias error in the predictionoutput of the first neural network model 302B. In certain scenarios, theremoval of unstructured or unwanted training data points may beperformed, at a pre-analysis phase (such as training phase) or beforethe first neural network model 302B may be used in a production phasefor prediction or the classification task.

At 308, the first plurality of training data points 302A may be updated.In accordance with an embodiment, the circuitry 202 may be configured toupdate the first plurality of training data points 302A. To update thefirst plurality of training data points 302A, the circuitry 202 mayremove the selected set of training data points (for example thetraining data point “T4” and the training data point “Tn” described at304) from the first plurality of training data points 302A to generate afourth plurality of training data points. The fourth plurality oftraining data points may include the training data points of the firstplurality of training data points 302A for which the impact scores (forexample average score) is more than the second impact score threshold orwhich meet the predefined selection criteria 304A.

At 310, the first neural network model 302B may be re-trained with thegenerated fourth plurality of training data points. In accordance withan embodiment, the circuitry 202 may be configured to re-train the firstneural network model 302B with the generated fourth plurality oftraining data points. To re-train the first neural network model 302B,the circuitry 202 may be configured to update one or more parameters ofeach node of the first neural network model 302B based on whether anoutput of the final layer for a given input (from the fourth pluralityof training data points) matches a correct result based on a lossfunction for the first neural network model 302B. The above process maybe repeated for the same or different inputs (or for certain epochs)till a minima of loss function may be achieved and a training error maybe minimized. Several methods for re-training the first neural networkmodel 302B are known in art, for example, gradient descent, stochasticgradient descent, batch gradient descent, gradient boost,meta-heuristics, and the like.

The re-training of the first neural network model 302B based on thefourth plurality of training data points may improve an existingperformance of the first neural network model 302B. This may be becausethe fourth plurality of training data points may include the remainingof the first plurality of training data points after the removal ofunwanted training data points. As the removed unwanted training datapoints may be less impactful (i.e. based on lower impact scores) forprediction of the output of the first neural network model 302B, theremoval of such unwanted data points may reduce the overfitting or biaserror in the prediction output of the first neural network model 302Bafter the re-training.

Although the diagram 300 is illustrated as discrete operations, such as302, 304, 306, 308, and 310, the disclosure is not so limited.Accordingly, in certain embodiments, such discrete operations may befurther divided into additional operations, combined into feweroperations, or eliminated, depending on the particular implementationwithout detracting from the essence of the disclosed embodiments.

FIG. 4 is a diagram that illustrates exemplary operations forre-training the first neural network model of FIG. 1 based on selectionof external data points, in accordance with an embodiment of thedisclosure. FIG. 4 is explained in conjunction with elements from FIGS.1,2 and 3. With reference to FIG. 4, there is shown a diagram 400 todepict exemplary operations from 402 to 410. The exemplary operationsillustrated in the diagram 400 may start at 402 and may be performed byany computing system, apparatus, or device, such as, by the electronicdevice 102 of FIG. 1 or the circuitry 202 of FIG. 2.

At 402, a first plurality of external data points 402A may be retrieved.In accordance with an embodiment, the circuitry 202 may be configured toretrieve the first plurality of external data points 402A from thedatabase 108. The first plurality of external data points 402A may bedifferent from a first plurality of training data points 412 on which afirst neural network model 404A (i.e. similar to the first neuralnetwork model 104 in FIG. 1 or the first neural network model 302B inFIG. 3) may be already trained. In some embodiments, the first neuralnetwork model 404A may be trained on the fourth plurality of trainingdata points, as described in FIG. 3 (at 310). In such case, the trainedfirst neural network model 404A may be already cleansed neural networkmodel from which the unwanted training data points (for example with lowimpact score) is already removed as described in FIG. 3.

In an example, a first external data point 402B of the first pluralityof external data points 402A is the image data. For example, the firstexternal data point 402B corresponds to an image of a first dog. Thecircuitry 202 of the electronic device 102 may be configured to retrievethe first plurality of external data points 402A that may include thefirst external data point 402B. The first neural network model 404A maybe trained on the first plurality of training data points 412 (such as afirst training data point 412A, a second training data point 412B, and athird training data point 412C) which may be, for example, differentimages (i.e. images of dog as shown in FIG. 4). It may be noted that onefirst external data point 402B shown in FIG. 4 is merely an example. Thefirst plurality of external data points 402A may include more than oneexternal data points, without a deviation from scope of the disclosure.

At 404, a first plurality of impact scores for each of the firstplurality of external data points 402A may be determined. In accordancewith an embodiment, the circuitry 202 may be configured to determine thefirst plurality of impact scores for each of the first plurality ofexternal data points 402A by an application of the first neural networkmodel 404A on each of the first plurality of external data points 402A.The first plurality of impact scores may indicate a first amount ofcontribution of each of the first plurality of training data points 412,(such as the first training data point 412A, the second training datapoint 412B, and the third training data point 412C) of the first neuralnetwork model 404A towards prediction of each of the first plurality ofexternal data points 402A. In an exemplary embodiment, an impact scorefor the first external data point 402B may indicate the first amount ofcontribution of each of the first plurality of training data points 412towards prediction of the first external data point 402B, to determinethe first plurality of impact scores for the first external data point402B. In an example, for three training data points (as shown in FIG.4), the electronic device 102 may determine three impact scores for thefirst external data point 402B.

In accordance with an embodiment, the impact score may indicate aninfluence of each of the first plurality of training data points 412 onthe prediction or classification of the first external data point 402B.In an embodiment, the first amount of contribution (i.e. impact score)may indicate a number of features of each of the first plurality oftraining data points 412 that may have contributed (or used by the firstneural network model 404A) towards the correct prediction of each of thefirst plurality of external data points, for example the first externaldata point 402B. The circuitry 202 may determine the impact score forthe first external data point 402B by comparison of the number offeatures in the first external data point 402B and the number offeatures in each of the first plurality of training data points 412. Inaccordance with an embodiment, the first training data point 412Acorresponds to an image of a second dog. The circuitry 202 may comparefeatures of the first dog in the first external data point 402B withcorresponding features of the second dog in the first training datapoint 412A. For example, the image of the first dog in the firstexternal data point 402B may depict that the first dog may havefeatures, such as, a large face, big ears, wide eyes, a small tail, around nose, and a small built. Further, the image of the first dog maybe a front profile of the first dog. Moreover, the image of the seconddog in the first training data point 412A may indicate that the seconddog may have features, such as, a medium-sized face, big ears, smalleyes, a fluffy medium tail, a round nose, and a small built.Furthermore, the image of the second dog may be of a front profile ofthe second dog. The circuitry 202 may compare the features of the firstdog and the features of the second dog and determine that the features,such as “the big ears, the round nose, the small built, and the frontprofile” may be common features (i.e. four features) in the firstexternal data point 402B and the first training data point 412A. In anexample, the circuitry 202 may determine an impact score 414A for thefirst external data point 402B as “0.6” with respect to the amount ofthe contribution or influence of the first training data point 412Atowards the prediction or classification of the first external datapoint 402B as a dog image. In other words, as more number of features ofthe first training data point 412A matches with the features of thefirst external data point 402B, the first neural network model 404A(i.e. controlled by the circuitry 202) may predict or classify the firstexternal data point 402B as the image of the dog.

Similarly, the circuitry 202 may further compare the features of thefirst dog in the first external data point 402B with correspondingfeatures of a third dog in the second training data point 412B. Theimage of the third dog in the second training data point 412B mayindicate that the third dog may have a small face, small ears, smalleyes, a small tail, the round nose and a large built. Furthermore, theimage of the third dog may be a side profile of a face of the third dog,as shown in FIG. 4. The circuitry 202 may compare the features of thefirst dog and the features of the third dog and determine that thefeatures, such as, “the small tail and the round nose” may be the commonfeatures (i.e. two features) in the first external data point 402B andthe second training data point 412B. In an example, the circuitry 202may determine an impact score 4148 for the first external data point402B as “0.4” (with respect to the amount of contribution of the secondtraining data point 412B), which may be less than the impact score 414Aas the number of common features between the first external data point402B and the first training data point 412A may be more than the numberof common features between the first external data point 402B and thesecond training data point 412B.

The circuitry 202 may further compare the features of the first dog inthe first external data point 402B with corresponding features of afourth dog in the third training data point 412C. The image of thefourth dog in the third training data point 412C may indicate that thefourth dog may have a large face, small ears, wide eyes, a small tail, atriangular nose, and a small built. Furthermore, the image of the fourthdog may be of a front profile of the fourth dog. The circuitry 202 maycompare the features of the first dog and the features of the fourthdog, and determine that the features, such as, “the large face, thesmall ears, the wide eyes, the small tail (at a different position thana tail of the first dog), the small built, and the front profile” may bethe common features (i.e. five features) in the first external datapoint 402B and the third training data point 412C. In an example, thecircuitry 202 may determine an impact score 414C for the first externaldata point 402B as “0.8” (with respect to the amount of contribution ofthe third training data point 412C), which may be more than the impactscore 414A and the impact score 414B. This may be because the number ofcommon features (i.e. five) between the first external data point 402Band the third training data point 412C may be more than the number ofcommon features (i.e. four) between the first external data point 402Band the first training data point 412A, and the number of commonfeatures (i.e. two) between the first external data point 402B and thesecond training data point 412B. Thus, the determined first plurality ofimpact scores by the circuitry 202 may include the impact score 414A,the impact score 414B, and the impact score 414C, for the first externaldata point 402B.

In accordance with an embodiment, the circuitry 202 is furtherconfigured to apply the mathematical function to the determined firstplurality of impact scores of the first external data point 402B. Theapplication of the mathematical function to the determined firstplurality of impact scores may be similar to the application of themathematical function to the determined third plurality of impactscores, as described for example, in FIG. 3. The circuitry 202 maydetermine a first impact score (i.e. one impact score) for the firstexternal data point 402B based on the application of the mathematicalfunction to the determined first plurality of impact scores of the firstexternal data point 402B. In an exemplary embodiment, the circuitry 202may apply the average function as the mathematical function on theimpact score 414A, the impact score 414B and the impact score 414C todetermine the first impact score for the first external data point 402B.In such a case, the circuitry 202 may apply the average function on thefirst plurality of impact scores “0.6, 0.4, and 0.8” to determine thefirst impact score as “0.6” for the first external data point 402B.Similarly, the circuitry 202 may determine the first impact score foreach of the first plurality of external data points 402A. In anotherexample, in case of the application of the maxima function on theplurality of impact scores, the first impact score for the firstexternal data point 402B is “0.8”.

In accordance with an embodiment, the circuitry 202 may be configured togenerate the prediction score for each of the first plurality ofexternal data points 402A. In some embodiments, the circuitry 202 may beconfigured to apply the first neural network model 404A on each of thefirst plurality of external data points 402A to generate the predictionscore for each of the first plurality of external data points 402A. Inan example, the first neural network model 404A may predict the firstexternal data point 402B accurately as an image of the dog, and theprediction score for the first external data point 402B may bedetermined as “0.9”. In another example, the first neural network model404A may predict another external data point of the first plurality ofexternal data points 402A inaccurately as an image of the dog, and theprediction score for the external data point may be determined as “0.1”.

At 406, the first set of external data points may be selected. Inaccordance with an embodiment, the circuitry 202 may be configured toselect the first set of data points from the first plurality of externaldata points 402A, based on the determination of the first plurality ofimpact scores for each of the first plurality of external data points402A. In some embodiments, the circuitry 202 may be configured to selectthe first set of external data points from the first plurality ofexternal data points 402A based on the determined first impact score andthe generated prediction score for each of the first plurality ofexternal data points 402A.

In accordance with an embodiment, the circuitry 202 may select the firstset of external data points based on the predefined selection criteria406A (shown in FIG. 4). In some embodiments, the circuitry 202 may beconfigured to select the first set of external data points from thefirst plurality of external data points 402A based on a first range ofimpact scores. In such a scenario, the first range of impact scores maybe the predefined selection criteria 406A. Thus, the first set ofexternal data points may include external data points that may have animpact score that may lie within the first range of impact scores. In anembodiment, the first range of impact scores may be between “0.5” and“1.0”. In such a case, the first external data point 402B may beselected in the first set of external data points, as the first impactscore of the first external data point 402B may be “0.6” (which lies inthe first range of impact scores). Similarly, other external data pointsin the first plurality of external data points 402A, whose first impactscore may lie in the first range of impact scores may be selected in thefirst set of external data points.

In accordance with another embodiment, the circuitry 202 may beconfigured to select the first set of external data points from thefirst plurality of external data points 402A based on a first impactscore threshold. In such a scenario, the first impact score thresholdmay be the predefined selection criteria 406A. Thus, each external datapoint in the first plurality of external data points 402A that may havean impact score above the first impact score threshold, may be selectedin the first set of external data points. In an embodiment, the firstimpact score threshold may be for example, “0.5”. In such a case, thefirst external data point 402B may be selected in the first set ofexternal data points, as the first impact score for the first externaldata point 402B is “0.6” (which lies above the first impact scorethreshold). The determination of the first impact score for the firstexternal data point 402B based on the average function is described, forexample, at 404 in FIG. 4. Similarly, other external data points in thefirst plurality of external data points 402A, whose first impact scoremay lie above the first impact score threshold may be selected in thefirst set of external data points.

In some embodiments, the predefined selection criteria 406A for theselection of the first set of external data points from the firstplurality of external data points 402A may be, for example, “n” numberof highest positive impact scores (such as Top-N), or “n” number ofhighest negative impact scores (such as Least-N) and so forth. Forexample, the first impact scores determined for the first plurality ofexternal data points 402A may be ordered (such an increasing ordecreasing order), to select the first set of external data points basedon the predefined selection criteria 406A (such as Top-N or Least N).For example, first two external data points in the ordered firstplurality of external data points 402A may be selected as the first setof external data points based on the predefined selection criteria 304A,such as Top-2. In an embodiment, the predefined selection criteria 406Afor the selection of the first set of external data points may include acombination of the first impact score and the prediction score for eachof the first plurality of external data points 402A. For example, thepredefined selection criteria 406A may include different scoresthresholds (such as the first impact score threshold and a predictionthreshold) to select the first set of external data points from thefirst plurality of external data points 402A.

In accordance with an embodiment, the circuitry 202 may be configured togenerate a second matrix representation 416 to depict the firstplurality of external data points 402A based on the first impact score(i.e. “IS”) and the prediction score (“PS”) as shown in FIG. 4. Thefirst impact score may be represented on an X-axis of the second matrixrepresentation 416, whereas the prediction score may be represented on aY-axis of the second matrix representation 416. The second matrixrepresentation 416 may include a first quadrant 416A, a second quadrant4168, a third quadrant 416C and a fourth quadrant 416D. In an exemplaryembodiment, the external data points of the first plurality of externaldata points 402A that may have the prediction scores between a range of“0.5” and “1.0” may be classified as the external data points with ahigh prediction score. Further, the external data points of the firstplurality of external data points 402A that may have the predictionscores between a range of “0” and “0.5” may be classified as theexternal data points with a low prediction score. Similarly, theexternal data points of the first plurality of external data points 402Athat may have the first impact score between a range of “0.5” and “1”may be classified as the external data points with a high first impactscore. Further, the external data points that may have the predictionscores between a range of “0” and “0.5” may be classified as theexternal data points with a low first impact score, as shown in thesecond matrix representation 416 in FIG. 4.

The external data points of the first plurality of external data points402A that lie in the first quadrant 416A may have the high predictionscore and the high first impact score. The external data points of thefirst plurality of external data points 402A that lie in the secondquadrant 416B may have the high prediction score and the low firstimpact score. The external data points of the first plurality ofexternal data points 402A that lie in the third quadrant 416C may havethe low prediction score and the low first impact score. The externaldata points of the first plurality of external data points 402A that liein the fourth quadrant 416D may have the low prediction score and thehigh first impact score, as shown in the second matrix representation416 in FIG. 4. In accordance with an embodiment, the circuitry 202 mayselect the external data points (i.e. from the first plurality ofexternal data points 402A) that may lie in the first quadrant 416A andthe fourth quadrant 416D, as the first set of external data points.

In accordance with an embodiment, the circuitry 202 may be configured toselect a particular predefined selection criteria 406A (such as impactrange based, impact threshold based, order based, prediction based, orcombination) based on different factors, such as, but not limited to,the real-time application, a type of the first plurality of externaldata points 402A, or a type of the first neural network model 404A.

At 408, the first plurality of training data points 412 may be updated.In accordance with an embodiment, the circuitry 202 may be configured toupdate the first plurality of training data points 412 with the selectedfirst set of external data points to generate a second plurality oftraining data points (not shown in FIG. 4). The generated secondplurality of training data points may include the first plurality oftraining data points 412 and the selected first set of external datapoints (i.e. selected at 406). Thus, the first plurality of trainingdata points 412 may be updated to include the selected first set ofexternal data points to generate the second plurality of training datapoints. The selected first set of external data points may be moreimpactful external data points (i.e. out of the first plurality ofexternal data points 402A) for which the amount of contribution orinfluence of the first plurality of training data points 412 (i.e. onwhich the first neural network model 404A is already trained) is high.Therefore, inclusion of such selected first set of external data points(as more impactful external data points) in the first plurality oftraining data points 412 to generate the second plurality of trainingdata points may further enhance the accuracy or prediction performanceof the first neural network model 404A.

At 410, the first neural network model 404A may be re-trained with thegenerated second plurality of training data points. In accordance withan embodiment, the circuitry 202 may be configured to re-train the firstneural network model 404A with the generated second plurality oftraining data points. The re-training of the first neural network model404A based on the generated second plurality of training data points maybe similar to the re-training of the first neural network model 302Bdescribed further, for example, in FIG. 3 (at 310).

Advantageously, the generated second plurality of training data pointsmay be the appropriate training data points to re-train the first neuralnetwork model 404A. In accordance with an embodiment, the first neuralnetwork model 404A may be re-trained for a plurality of classes, such asfor a class dog, a class cat, a class horse and so forth. In such acase, the circuitry 202 may enable efficient selection of appropriateexternal data points from a large external dataset (i.e. external data)of the plurality of classes. The first plurality of external data points402A (which may be unknown to the first neural network model 404A) maythereby prioritized for the selection of the first set of external datapoints for the re-training by the disclosed electronic device 102,instead of usage of all external data points in the first plurality ofexternal data points 402A for the re-training. Further, the first set ofexternal data points may be more effective or impactful for there-training of the first neural network model 404A, as such externaldata points may be selected based on the determined impact scores foreach of the first plurality of external data points 402A, instead ofrandom selection. Thus, the disclosed electronic device 102 may therebyenable efficient and faster re-training of the first neural networkmodel 404A based on prioritization (or selection) of impactful externaldata points from the first plurality of external data points 402A,instead of usage of random external data points or all external datapoints of the first plurality of external data points 402A for there-training.

In accordance with an embodiment, the circuitry 202 may be furtherconfigured to apply the re-trained first neural network model 404A onthe selected first set of external data points (on which the firstneural network model 404A is re-trained at 410), to generate a set ofprediction scores for the validation of the prediction for the first setof external data points. The validation of the prediction for the firstset of external data points by the generated set of prediction scoresmay further verify that the performance of the re-trained first neuralnetwork model 404A is better (or not) than the performance of the firstneural network model 404A before re-training. In some embodiments, thecircuitry 202 may be further configured to apply the re-trained firstneural network model 404A on other external data points (different thanthe selected first set of external data points) to generate the set ofprediction scores for the validation of the prediction for the first setof external data points.

The circuitry 202 may be configured to generate the second graphicalrepresentation. The second graphical representation may be similar tothe first graphical representation but may be for each external datapoint in the first plurality of external data points 110, instead ofeach training data point in the first plurality of training data points302A. For example, the second graphical representation may include aplurality of nodes and a plurality of edges between the plurality ofnodes. Few nodes of the plurality of nodes in the second graphicalrepresentation may represent an external data point of the firstplurality of external data points 110 and other nodes in the pluralityof nodes may represent a training data point. Each edge of the pluralityof edges in the second graphical representation may represent an impactscore (or a weightage value) of a plurality of impact scores for eachexternal data point. The impact score may indicate an amount ofcontribution of each of the first plurality of training data points 106of the first neural network model 302B towards a prediction of theexternal data point of the first plurality of external data points 110as described, for example in FIGS. 1 and 4.

The circuitry 202 may be configured to display the determined graphicalrelations (i.e., the first graphical representation 302C and/or thesecond graphical representation) through the I/O device 206. This mayenable a user of the electronic device 102 to understand impactfulexternal data points (as well as training data points) and alsounderstand a feature extraction ability of the trained first neuralnetwork model 302B for the prioritization and selection or removal ofdata points (e.g., the first plurality of training data points 106and/or the first plurality of external data points 110).

Although the diagram 400 is illustrated as discrete operations, such as402, 404, 406, 408, and 410, the disclosure is not so limited.Accordingly, in certain embodiments, such discrete operations may befurther divided into additional operations, combined into feweroperations, or eliminated, depending on the particular implementationwithout detracting from the essence of the disclosed embodiments.

FIG. 5 is a diagram that illustrates exemplary operations forre-training the first neural network model based on selection ofexternal data points with applied plurality of realistic variations, inaccordance with an embodiment of the disclosure. FIG. 5 is explained inconjunction with elements from FIGS. 1, 2, 3, and 4. With reference toFIG. 5, there is shown a diagram 500 to depict exemplary operations from502 to 512. The exemplary operations illustrated in the diagram 500 maystart at 502 and may be performed by any computing system, apparatus, ordevice, such as by the electronic device 102 of FIG. 1 or the circuitry202 of FIG. 2.

At 502, a plurality of realistic variations may be applied to one ormore external data points of a first plurality of external data points502A. In accordance with an embodiment, the circuitry 202 may beconfigured to apply the plurality of realistic variations to the one ormore external data points, (for example, an external data point 514 ofthe first plurality of external data points 502A) to generate a secondplurality of external data points, such as the second plurality ofexternal data points 516. The one or more external data points, such asa first external data point 516A, a second external data point 516B, athird external data point 516C, and a fourth external data point 516D ofthe generated second plurality of external data points 516 maycorrespond to different realistic variation of the plurality ofrealistic variations applied on the external data point 514 of the firstplurality of external data points 502A. It may be noted that theexternal data point 514 shown as a dog image in FIG. 5, is merelypresented as an example. The external data point 514 may be any imagedata, audio data, text data, three-dimensional (3D) data, or electricalsignals, without a deviation from the scope of the disclosure.

In accordance with an embodiment, the plurality of realistic variationsmay include, but is not limited to, a rotation variation, a horizontaltranslation variation, a vertical translation variation, a shearvariation, a zoom variation, a brightness variation, a contrastvariation, a flip variation, a sharpness variation, or a colorvariation. In an exemplary implementation, the first external data point516A may be a flip variation of the external data point 514, and thesecond external data point 516B may be a zoom variation (e.g., azoomed-out variation) of the external data point 514. Further, the thirdexternal data point 516C may be a zoom variation (e.g., a zoomed-invariation) of the external data point 514, and the fourth external datapoint 516D may be a rotation variation of the external data point 514.In an example, the one or more external data points may be the audiodata. In such a case, the plurality of realistic variations may include,but not limited to, audio augmentation techniques, such asincrease/decrease in volume of the audio data (i.e. volume variation),change in frequency of the audio data (i.e. frequency variation), changein sound equalization (i.e. audio equitization variation), tonevariation, and/or addition of noise in the audio data (i.e. audio noisevariation).

At 504, a second plurality of impact scores for each of the generatedsecond plurality of external data points 516 may be determined. Inaccordance with an embodiment, the circuitry 202 may be configured toapply the first neural network model 404A on the generated secondplurality of external data points 516 to determine the second pluralityof impact scores for each of the generated second plurality of externaldata points 516. The second plurality of impact scores may indicate asecond amount of contribution or influence of each of the firstplurality of training data points 412 (shown in FIG. 4) of the firstneural network model 404A towards a prediction of each of the generatedsecond plurality of external data points 516. The determination of thesecond plurality of impact scores for the second plurality of externaldata points 516 is similar to the determination of the first pluralityof impact scores for the first plurality of external data points 502A,as described, for example, in FIG. 4 at 404.

In an exemplary implementation, the circuitry 202 may determine animpact score 518A as “0.9” for the first external data point 516A basedon the comparison of features of the external data point 514 withcorresponding features of the first external data point 516A. As thefirst external data point 516A is a flip variation of the external datapoint 514, a majority of features may be common between the externaldata point 514 and the first external data point 516A. In other words,an amount of contribution or influence of at least one of the firstplurality of training data points 412 (i.e. on which the first neuralnetwork model 404A is trained) towards the prediction of the firstexternal data point 516A or the external data point 514 may high (i.e.as described, for example, in FIG. 4). Thus, the circuitry 202 maydetermine the impact score 518A, for example, as “0.9” for the firstexternal data point 516A. As another example, the circuitry 202 mayfurther determine an impact score 518B as “0.2” for the second externaldata point 516B based on the comparison of the features of the externaldata point 514 with corresponding features of the second external datapoint 516B. As the second external data point 516B is a zoomed-outvariation of the external data point 514, there may be a few commonfeatures between the external data point 514 and the second externaldata point 516B or few common features between the second external datapoint 516B and at least one of the first plurality of training datapoints 412. Thus, the circuitry 202 may further determine an impactscore 518B as “0.2” for the second external data point 516B. Similarly,the circuitry 202 may determine an impact score 518C as “0.4” for thethird external data point 516C (i.e. a zoomed-in variation) and animpact score 518D as “0.7” for the fourth external data point 516D (i.e.rotation variation).

At 506, a second set of external data points may be selected from thegenerated second plurality of external data points 516. In accordancewith an embodiment, the circuitry 202 may be configured to select thesecond set of external data points from the generated second pluralityof external data points 516 based on the determined second plurality ofimpact scores for each of the generated second plurality of externaldata points 516. In some embodiments, the circuitry 202 may select thesecond set of external data points from the generated second pluralityof external data points 516 based on a predefined selection criteria,such as, the predefined selection criteria described, for example, inFIGS. 3 and 4. In an example, the circuitry 202 may select the secondset of external data points based on the predefined selection criteria,such as, based on a range of the impact scores. For example, thecircuitry 202 may select external data points from the second pluralityof external data points 516, whose impact score lie between the range of“0.5” and “1.0”. Thus, the circuitry 202 may select the first externaldata point 516A with the impact score 518A and the fourth external datapoint 516D with the impact score 518D, as the second set of externaldata points from the generated second plurality of external data points516. In such scenario, a first realistic variation (e.g., the flipvariation), applied to the external data point 514 to generate the firstexternal data point 516A, may be different from a second realisticvariation (e.g., the rotation variation), applied to the external datapoint 514 to generate the fourth external data point 516D. However, thescope of the disclosure may not be limited to the fact that the externaldata points (such as the first external data point 516A and the fourthexternal data point 516D) selected in the second set of external datapoints, may have different realistic variations, such as the firstrealistic variation is different from the second realistic variation. Incertain embodiments, the first realistic variation and the secondrealistic variation of different selected external data points may besame, without departure from the scope of the disclosure.

At 508, the first plurality of training data points, such as, the firstplurality of training data points 412 may be updated. The circuitry 202may be configured to update the first plurality of training data points412 with the second set of external data points selected from thegenerated second plurality of external data points 516, to generate athird plurality of training data points. The third plurality of trainingdata points may include the first plurality of training data points 412and the selected second set of external data points. Thus, the circuitry202 may select external data points, such as, the first external datapoint 516A and the fourth external data point 516D from the secondplurality of external data points 516, as augmented external datapoints. The augmented external data points may be added to the firstplurality of training data points 412 (i.e., an original training datapoints) to re-train the first neural network model 404A and enhance aperformance of the first neural network model 404A. With reference toFIG. 3, the augmented external data points may be added to the fourthplurality of training data points (i.e., cleaned training data points)to re-train the first neural network model 404A and enhance aperformance of the first neural network model 404A.

At 510, a realistic variation may be selected from the plurality ofrealistic variations applied to the one or more external data points ofthe first plurality of external data points 502A. In accordance with anembodiment, the circuitry 202 may be configured to analyze the realisticvariations of the external data points in the selected second set ofexternal data points, to determine the most common realistic variationin the selected second set of external data points (i.e. with higherimpact scores). The circuitry 202 may select a realistic variation (asmost common) from the plurality of realistic variations, for futureapplication (as shown in FIG. 5) of the realistic variations on thefirst plurality of external data points 502A (i.e. described, forexample, at 502), since the selected realistic variation may be a bestvariation with higher probability in the selection of external datapoints and to increase performance of the first neural network model404A. Thus, the selected realistic variation may provide more impactfulexternal data points on which the first neural network model 404A can befurther re-trained to increase the accuracy of the prediction orclassification task. For example, in case the flip variation provides ahigh impact score (i.e. which may satisfy the predefined selectioncriteria) to the one or more selected external data points of the firstplurality of external data points 502A, the circuitry 202 may select theflip variation as the appropriate realistic variation. The circuitry 202may apply such flip variation (as the selected realistic variation) tothe one or more external data points of the first plurality of externaldata points 502A to generate the second plurality of external datapoints 516 in future. Thus, the circuitry 202 may thereby prioritizeaugmentation techniques (or realistic variations) and select anappropriate augmentation technique to apply on one or more of the firstplurality of external data points 502A to generate the augmented datapoints based on impact scores determined for each augmentation techniquewith respect to the first neural network model 404A. In an embodiment,different realistic variations (or augmentation techniques) may beapplied to different types of images in the one or more external datapoints based on impact scores associated with the variations on theimages. For example, a flip variation may provide a high impact scorefor one or more first external data points (e.g., images of dogs), whilea rotation variation may provide a high impact score for one or moresecond external data points (e.g., images of cats or other types ofimages or data points). In such case, the circuitry 202 may select theflip variation as the appropriate realistic variation for the one ormore first external data points and select the rotation variation as theappropriate realistic variation for the one or more second external datapoints.

At 512, the first neural network model 404A may be re-trained with thegenerated third plurality of training data points. In accordance with anembodiment, the circuitry 202 may be configured to re-train the firstneural network model 404A with the generated third plurality of trainingdata points (i.e. generated at 508). The re-training of the first neuralnetwork model 404A based on the generated third plurality of trainingdata points may be similar to the re-training of the first neuralnetwork model 404A described further, for example, in FIG. 3 (at 310)and FIG. 4 (at 410). The re-training of the first neural network model404A based on the addition of the augmented external data points to theoriginal training data points may make the first neural network model404A robust to the variations in external data points, as this may leadto an increase in the diversity of a training data set for there-training and further improves an accuracy of prediction output of thefirst neural network model 404A. Further, as the augmented external datapoints may include most impactful (based on high impact scores)realistic variations of external data points, the augmented externaldata points may be more useful for the re-training of the first neuralnetwork model 404A, instead of realistic variations of external datapoints selected randomly or based on trial and error.

Although the diagram 500 is illustrated as discrete operations, such as502, 504, 506, 508, 510, and 512, the disclosure is not so limited.Accordingly, in certain embodiments, such discrete operations may befurther divided into additional operations, combined into feweroperations, or eliminated, depending on the particular implementationwithout detracting from the essence of the disclosed embodiments.

FIG. 6 is a block diagram that illustrates exemplary dynamic tracking ofthe first neural network model, in accordance with an embodiment of thedisclosure. FIG. 6 is explained in conjunction with elements from FIGS.1, 2, 3, 4, and 5. With reference to FIG. 6, there is shown a blockdiagram 600. The block diagram 600 may include an electronic device 602.In one or more embodiments, the electronic device 602 may be similar tothe electronic device 102. The electronic device 602 may include a firstneural network model 604 (i.e. similar to the first neural network model104 in FIG. 1). The block diagram 600 may further include a firstplurality of training data points 606 on which the first neural networkmodel 604 has to be trained or retrained. In such case the firstplurality of training data points 606 may be similar to the secondplurality of training data points (i.e. described at 408-410 in FIG. 4),or similar to the third plurality of training data points (i.e.described at 508 and 510 in FIG. 5), or similar to the fourth pluralityof training data points (i.e. described at 308-310 in FIG. 3). In someembodiments, the first plurality of training data points 606 may beoriginal training data points to train an un-trained neural networkmodel. The circuitry 202 may dynamically track the training of the firstneural network model 604 for a plurality of epochs, such as a predefinednumber of epochs (for example 100 in number) for which first neuralnetwork model 604 has to be trained. The plurality of epochs may includefirst set of epochs 608A, a second set of epochs 608B, . . . and an Nthset of epochs 608N.

The circuitry 202 may be further configured to select the first neuralnetwork model 604 for training for the plurality of epochs based on thefirst plurality of training data points 606. The circuitry 202 may befurther configured to control a set of operations for training theselected first neural network model 604. The set of operations mayinclude control of the training of the first neural network model 604for one of more epochs (such as the first set of epochs 608A of theplurality of epochs) to generate a second neural network model trainedon the first plurality of training data points 606 for the first set ofepochs 608A which may be a sub-set of the plurality of epochs. Thecircuitry 202 of the electronic device 602 may be configured to controlthe training of the first neural network model 604 for the first set ofepochs 608A of the plurality of epochs to generate the second neuralnetwork model.

The circuitry 202 may be further configured to determine an impact scorefor each of the first plurality of training data points 606 during thetraining of the first neural network model 604 for the first set ofepochs 608A. The impact score may indicate an amount of contribution (orimpact/influence) of each of the first plurality of training data points606 of the generated second neural network model towards prediction ofeach of the first plurality of training data points 606 (as described,for example, in FIG. 3 at 302). For the determination of the impactscore, the circuitry 202 may determine a first set of impact scores 610Afor the first plurality of training data points 606 based on thecomparison of features in each of the first plurality of training datapoints 606 with corresponding features in each of other training datapoints of the first plurality of training data points 606.Thedetermination of the impact scores (i.e. also referred as data selectionplug-in) for the first plurality of training data points 606 during thetraining of the first neural network model 604 may be referred asdynamic tracking of the training of the first neural network model 604.

The circuitry 202 may be further configured to re-select the generatedsecond neural network model as the first neural network model 604 forthe training for others (i.e. remaining) of the plurality of epochs,based on a comparison between the first set of impact scores 610A and atraining impact threshold. In an example, the training impact thresholdmay be a value between “0” and “1”, such as, “0.55”. In an exemplaryembodiment, in case the first set of impact scores 610A (i.e. each scoreor average score) for the first plurality of training data points 606,after the training for the first set of epochs 608A, is less than thetraining impact threshold (e.g., 0.55), the circuitry 202 may beconfigured to re-select the generated second neural network model as thefirst neural network model 604 for further training (say for the secondset of epochs 608B). In another exemplary embodiment, in case an averageimpact score of the first set of impact scores 610A after the trainingfor the first set of epochs 608A is less than the training impactthreshold, then the circuitry 202 may be configured to re-select thegenerated second neural network model as the first neural network model604 for further training (say for the second set of epochs 608B).Alternatively, in case the first set of impact scores 610A (or theaverage score) is more than the training impact threshold (e.g., 0.55),the circuitry 202 may obtain the generated second neural network modelas the trained second neural network model or a best neural networkmodel 612 as shown in FIG. 6. In such case, the electronic device 602may stop further training of the neural network model (say for thesecond set of epochs 608B and the Nth set of epochs 608N) and thegenerated second neural network model (i.e. intermediate model) may beobtained as a final or best neural network model 612.

In an embodiment, in case the first set of impact scores 610A for thefirst set of epochs 608A is less than the training impact threshold, thecircuitry 202 may be configured to control the training of the generatedsecond neural network model (as the re-selected the first neural networkmodel 604) for the second set of epochs 608B of the plurality of epochsto generate a third neural network model. In other words, the circuitry202 may control the second neural network model to continue training forthe other sets of epochs of the plurality of epochs (such as the secondset of epochs 608B and the and the Nth set of epochs 608N). Similarly,the circuitry 202 may be configured to re-select the generated thirdneural network model as the first neural network model 604, in case thesecond set of impact scores 610B during/after the training for thesecond set of epochs 608B is less than the training impact threshold.Alternatively, the circuitry 202 may be configured to obtain thegenerated third neural network model as the trained third neural networkmodel or the best neural network model 612, in case the second set ofimpact scores 6108 (or the average score) is more than the trainingimpact threshold. Thus, the circuitry 202 may be configured to obtainthe best neural network model 612 based on an iterative control of theset of operations and dynamically track of the impact scores for thefirst plurality of training data points 606 after training the neuralnetwork model for one or more epochs (i.e. subset of the plurality ofepochs). The electronic device 602 may thereby ensure that the bestneural network model 612 may be selected based on the dynamic trackingof the training of the first neural network model 604. Thus, based onthe dynamic determination of impact scores for training data points ineach set of epochs, the first neural network model 604 may beeffectively trained and the best neural network model 612 may beselected at intermediate training stage when the impact scores for acertain set of epochs become greater than the training impact threshold,rather than training the first neural network model 604 for the completeplurality of epochs (i.e. which may be large number of epochs). This mayreduce the time and computational resources required for the training ofthe first neural network model 604, as the best neural network model 612may be obtained faster and in a lesser number of epochs.

FIGS. 7A-7B are block diagrams that illustrates exemplary operations ofthe electronic device to control a first neural network model and aplurality of second neural network models, in accordance with anembodiment of the disclosure. FIGS. 7A-7B are explained in conjunctionwith elements from FIGS. 1, 2, 3, 4, 5, and 6. With reference to FIG.7A, there is shown a block diagram 700A. The block diagram 700A mayinclude the electronic device 102 and a first neural network model 702in a production phase 704 of a particular application (for examplereal-time application described in FIG. 3). The production phase 704 maycorrespond to a phase in a real-time production environment, where thetrained neural network models (such as the first neural network model702) may be deployed to perform prediction (such as the classificationtask) for the real-time application. There is further shown a pluralityof second neural network models 708A-708N that may be different from thefirst neural network model 702. For example, each of the first neuralnetwork model 702 and the plurality of second neural network models708A-708N may be trained on different types of training data points (forexample trained on different labels of images or trained on differentlabels of other data points like audio data, text data). The pluralityof second neural network models 708A-708N may include a first neuralnetwork model 708A, a second neural network model 708B, a third neuralnetwork model 708C, a fourth neural network model 708D, . . . and an Nthneural network model 708N. As shown in FIG. 7A, the production phase 704also include the first neural network model 708A and the second neuralnetwork model. There is further shown in FIG. 7A, a staging phase 706that may include the third neural network model 708C, the fourth neuralnetwork model 708D, . . . and the Nth neural network model 708N. In anembodiment, the staging phase 706 may correspond to a phase in which oneor more neural network models (e.g., the third neural network model708C, the fourth neural network model 708D, . . . and the Nth neuralnetwork model 708N) may be tested prior to the deployment in thereal-time production environment. The staging phase 706 may enable adetermination of a prediction performance of such one or more neuralnetwork models and to select suitable neural network model(s) for thedeployment in the real-time production environment (i.e., for theproduction phase 704). The block diagram 700A may further include asecond external data point 710 (for example similar to the firstexternal data point 402B in FIG. 4).

In accordance with an embodiment, the circuitry 202 of the electronicdevice 102 may be configured to apply the first neural network model 702and the plurality of second neural network models 708A-708N on thesecond external data point 710. The first neural network model 702 maybe trained on the first plurality of training data points 106. Each ofthe plurality of second neural network models 708A-708N may be trainedon a plurality of training data points which may be same or differentfrom the first plurality of training data points 106. The circuitry 202may be further configured to determine a plurality of impact scores forthe second external data point 710 based on the application of each ofthe first neural network model 702 and the plurality of second neuralnetwork models 708A-708N on the second external data point 710. Theplurality of impact scores for the second external data point 710 mayindicate an amount of contribution of the first plurality of trainingdata points 106 (of the first neural network model 702) and theplurality of training data points (of each of the plurality of secondneural network models 708A-708N) towards prediction of the secondexternal data point 710. The circuitry 202 may determine the pluralityof impact scores for the second external data point 710 based on thecomparison of features of the second external data point 710 and withcorresponding features of training data points, on which the firstneural network model 702, and the plurality of second neural networkmodels 708A-708N may be trained (as also described, for example, inFIGS. 3-4).

The circuitry 202 may be further configured to switch a neural networkmodel from the production phase 704 with a neural network model in thestaging phase 706 during the runtime usage of the first neural networkmodel 702. In an exemplary implementation, the circuitry 202 may switchthe first neural network model 702 in the production phase 704 with thethird neural network model 708C (as a selected neural network model inthe staging phase 706), based on a determination that an impact scorefor the second external data point 710 with respect to the third neuralnetwork model 708C is more than an impact score for the second externaldata point 710 with respect to the first neural network model 702. Thecircuitry 202 may be further configured to control the memory 204 of theelectronic device 102 to store the selected neural network model, suchas the third neural network model 708C, in the memory 204 of theelectronic device 102. In some embodiments, the circuitry 202 mayutilize a model impact threshold to control the switching of neuralnetwork model between the production phase 704 and the staging phase706.

The circuitry 202 may switch a neural network model from the stagingphase 706 to the production phase 704 based on a determination that animpact of the neural network model for a certain external data point isgreater than an impact of a counter-part neural network model in theproduction phase 704, for the same external data point. Thus, as theswitching or movement between the neural network model and thecounter-part neural network model is based on the impact determinationof the respective neural network models for the same external datapoint, the production phase 704 may include neural network models thatmay be capable of more accurate prediction for the particular externaldata point.

With reference to FIG.7B, there is shown a block diagram 700B that mayinclude the electronic device 102 The production phase 704 may includethe first neural network model 702, and further include the first neuralnetwork model 708A, and the second neural network model 708B of theplurality of second neural network models 708A-708N. The staging phase706 may include the third neural network model 708C, the fourth neuralnetwork model 708D, . . . and the Nth neural network model 708N, similarto FIG. 7A.

The block diagram 700B represents a selection of a new neural networkmodel from the staging phase 706 for addition in the production phase704 at a runtime usage of the first neural network model 702. Inaccordance with an embodiment, the circuitry 202 may be configured toselect a neural network model, such as the third neural network model708C, from the staging phase 706 and add the selected neural networkmodel in the production phase 704. The selection of the neural networkmodel and the addition of the selected neural network model to theproduction phase 704 may be based on a determination of a plurality ofimpact scores for the second external data point 710 with respect toeach of the plurality of second neural network models 708A-708N. Inaccordance with an embodiment, the circuitry 202 of the electronicdevice 102 may be configured to apply the plurality of second neuralnetwork models 708A-708N on the second external data point 710. Aspreviously discussed, each of the plurality of second neural networkmodels 708A-708N may be trained on a plurality of training data points.The circuitry 202 may be further configured to determine the pluralityof impact scores for the second external data point 710 based on theapplication of each of the plurality of second neural network models708A-708N on the second external data point 710. The plurality of impactscores for the second external data point 710 may indicate an amount ofcontribution of the plurality of training data points (of each of theplurality of second neural network models 708A-708N) towards predictionof the second external data point 710. The circuitry 202 may determinethe plurality of impact scores for the second external data point 710based on the comparison of features of the second external data point710 and with corresponding features of training data points, on whichthe plurality of second neural network models 708A-708N may be trained.The circuitry 202 may further select and add the third neural networkmodel 708C to the production phase 704, based on the determination thatthe impact score of the third neural network model 708C is more thancorresponding impact scores of the fourth neural network model 708D, . .. and the Nth neural network model 708N. The circuitry 202 may furthercontrol the memory 204 of the electronic device 102 to store theselected third neural network model 708C in the production phase 704along with the first neural network model 702, and the first neuralnetwork model 708A, and the second neural network model 708B of theplurality of second neural network models 708A-708N.

The circuitry 202 may select a neural network model from a group ofneural network models present in the staging phase 706 based on adetermination that an impact score of the selected neural network modelfor a certain external data point is greater than an impact score ofeach of the remaining neural network models in the staging phase 706,for the same external data point. The selected neural network model maythus be a most impactful neural network model for the external datapoint with respect to the other neural network models in the group ofneural network models present in the staging phase 706. After theaddition of the new neural network model to the production phase 704,the production phase 704 may include neural network models that may becapable of more accurate prediction for the external data points.

FIG. 8 is a flowchart that illustrates an exemplary method forre-training of the first neural network model, based on training datapoints, in accordance with an embodiment of the disclosure. FIG. 8 isdescribed in conjunction with elements from FIGS. 1, 2, 3, 4, 5, 6, 7A,and 7B. With reference to FIG. 8, there is shown a flowchart 800. Theoperations of the flowchart 800 may be executed by a computing system,such as the electronic device 102 or the circuitry 202. The operationsmay start at 802 and proceed to 804.

At 804, the first neural network model 302B trained for a classificationtask of a real-time application may be stored. In one or moreembodiments, the circuitry 202 of the electronic device 102 may beconfigured to store the first neural network model 302B in a memory(such as, the memory 204) of the electronic device 102, as described,for example, in FIG. 2. The first neural network model 302B may betrained with a first plurality of training data points (such as, thefirst plurality of training data points 106).

At 806, the third plurality of impact scores for each training datapoint of the first plurality of training data points 302A may bedetermined. In accordance with an embodiment, the circuitry 202 of theelectronic device 102 may be configured to determine the third pluralityof impact scores for each training data point of the first plurality oftraining data points 302A, based on the application of the first neuralnetwork model 302B on the first plurality of training data points 302A,as described, for example, in FIG. 3 (at 302). The third plurality ofimpact scores may indicate the third amount of contribution of each ofthe first plurality of training data points 302A of the first neuralnetwork model 302B towards prediction of each of the first plurality oftraining data points 302A.

At 808, the set of training data points may be selected from the firstplurality of training data points 302A, based on the determined thirdplurality of impact scores for each of the first plurality of trainingdata points 302A. In accordance with an embodiment, the circuitry 202 ofthe electronic device 102 may be configured to select the set oftraining data points from the first plurality of training data points302A based on the determined third plurality of impact scores for eachof the first plurality of training data points 302A, as described, forexample, in FIG. 3 (at 304).

At a 810, the first plurality of training data points 302A may beupdated with removal of the selected set of training data points togenerate the fourth plurality of training data points. In accordancewith an embodiment, the circuitry 202 of the electronic device 102 maybe configured to update the first plurality of training data points 302Awith removal of the selected set of training data points to generate thefourth plurality of training data points, as described, for example, inFIG. 3 (at 306 and 308).

At 812, the first neural network model 302B may be re-trained with thegenerated fourth plurality of training data points. In accordance withan embodiment, the circuitry 202 of the electronic device 102 may beconfigured to re-train the first neural network model 302B with thegenerated fourth plurality of training data points, described, forexample, in FIG. 3 (at 310). Control may pass to end.

Although the flowchart 800 is illustrated as discrete operations, suchas 804, 806, 808, 810 and 812, the disclosure is not so limited.Accordingly, in certain embodiments, such discrete operations may befurther divided into additional operations, combined into feweroperations, or eliminated, depending on the particular implementationwithout detracting from the essence of the disclosed embodiments.

FIG. 9 is a flowchart that illustrates an exemplary method forre-training of the first neural network model, in accordance with anembodiment of the disclosure. FIG. 9 is described in conjunction withelements from FIGS. 1, 2, 3, 4, 5, 6, 7A, 7B, and 8. With reference toFIG. 9, there is shown a flowchart 900. The operations of the flowchart900 may be executed by a computing system, such as the electronic device102 or the circuitry 202. The operations may start at 902 and proceed to904.

At 904, the first neural network model 404A trained for a classificationtask of a real-time application may be stored. In one or moreembodiments, the circuitry 202 of the electronic device 102 may beconfigured to store the first neural network model 404A in a memory(such as, the memory 204) of the electronic device 102, as described,for example, in FIG. 2. The first neural network model 104 may betrained with a first plurality of training data points (such as, thefirst plurality of training data points 106).

At 906, the first plurality of external data points 402A may beretrieved. The first plurality of external data points 402A (or thefirst plurality of external data points 110) may be different from thefirst plurality of training data points 412 on which the first neuralnetwork model 404A may be trained. In one or more embodiments, thecircuitry 202 of the electronic device 102 may be configured to retrievethe first plurality of external data points 402A (i.e. external data)from the database 108, as described, for example, in FIG. 4 (at 402).

At 908, the first neural network model 404A may be applied on the firstplurality of external data points 402A to determine a first plurality ofimpact scores for each of the first plurality of external data points402A. In one or more embodiments, the circuitry 202 of the electronicdevice 102 may be configured to apply the first neural network model404A on the first plurality of external data points 402A to determinethe first plurality of impact scores (such as, the impact score 414A,the impact score 414B and the impact score 414C) for each of the firstplurality of external data points 402A, as described, for example, inFIG. 4 (at 404). The first plurality of impact scores may indicate afirst amount of contribution or impact/influence of each of the firstplurality of training data points 412 of the first neural network model404A towards prediction of each of the first plurality of external datapoints 402A.

At 910, a first set of external data points may be selected from thefirst plurality of external data points 402A based on the determinedfirst plurality of impact scores for each of the first plurality ofexternal data points 402A. In one or more embodiments, the circuitry 202of the electronic device 102 may be configured to select the first setof external data points (such as the first external data point 402B)from the first plurality of external data points 402A based on thedetermined first plurality of impact scores for each of the firstplurality of external data points 502A, as described, for example, inFIG. 4 (at 406).

At 912, the first plurality of training data points 412 may be updatedwith the selected first set of external data points (such as, the firstexternal data point 402B) to generate a second plurality of trainingdata points. In one or more embodiments, the circuitry 202 of theelectronic device 102 may be configured to update the first plurality oftraining data points 412 by addition of the selected first set ofexternal data points (such as, the first external data point 402B) togenerate a second plurality of training data points, as described, forexample, in FIG. 4 (at 408).

At 914, the first neural network model 404A may be re-trained with thegenerated second plurality of training data points. In one or moreembodiments, the circuitry 202 of the electronic device 102 may beconfigured to re-train the first neural network model 404A with thegenerated second plurality of training data points, as described, forexample, in FIG. 4 (at 410). Control may pass to end.

Although the flowchart 900 is illustrated as discrete operations, suchas 904, 906, 908, 910, 912 and 914, the disclosure is not so limited.Accordingly, in certain embodiments, such discrete operations may befurther divided into additional operations, combined into feweroperations, or eliminated, depending on the particular implementationwithout detracting from the essence of the disclosed embodiments.

FIG. 10 is a flowchart that illustrates an exemplary method for dynamictracking of the first neural network model, in accordance with anembodiment of the disclosure. FIG. 10 is described in conjunction withelements from FIGS. 1, 2, 3, 4, 5, 6, 7A, 7B, 8, and 9. With referenceto FIG. 10, there is shown a flowchart 1000. The operations of theflowchart 1000 may be executed by a computing system, such as theelectronic device 102 or the circuitry 202. The operations may start at1002 and proceed to 1004.

At 1004, a first neural network model may be selected for training for aplurality of epochs based on a first plurality of training data points.In accordance with an embodiment, the circuitry 202 may be configured toselect the first neural network model 604 for training for the pluralityof epochs, (such as, the first set of epochs 608A and the second set ofepochs 608B) based on the first plurality of training data points 606,as described, for example, in FIG. 6.

At 1006, the training of the first neural network model 604 may becontrolled for one or more epochs of the plurality of epochs to generatea second neural network model trained on the first plurality of trainingdata points 606 for the one or more epochs. In accordance with anembodiment, the circuitry 202 may be configured to control the trainingof the first neural network model 604 for the one or more epochs, (suchas the first set of epochs 608A of the plurality of epochs), to generatethe second neural network model trained on the first plurality oftraining data points 606 for the one or more epochs. The control of thetraining of the first neural network model 604 for the one or moreepochs to generate the second neural network model, is described, forexample, in FIG. 6.

At 1008, an impact score may be determined for each of the firstplurality of training data points 606. The impact score may indicate anamount of contribution (or impact) of each of the first plurality oftraining data points 606 of the generated second neural network modeltowards a prediction of each of the first plurality of training datapoints 606. In accordance with an embodiment, the circuitry 202 may beconfigured to determine the impact score, such as the first set ofimpact scores 610A, for each of the first plurality of training datapoints 606 for the first set of epochs 608A. The determination of theimpact score for each of the first plurality of training data points606, is described, for example, in FIG. 6.

At 1010, it may be determined whether the determined impact score (at1008) is greater than a training impact threshold. In accordance with anembodiment, the circuitry 202 may be configured to determine whether thedetermined impact score, (such as the first set of impact scores 610A),is more than the training impact threshold or not, as described, forexample, in FIG. 6. In case it is determined that the determined impactscore is greater than the training impact threshold, control may pass to1014. Otherwise, control may pass to 1012.

At 1012, the generated second neural network model may be re-selected asthe first neural network model 604 for training for others of theplurality of epochs, if the first set of impact scores 610A is less thanthe training impact threshold. In accordance with an embodiment, thecircuitry 202 may be configured to re-select the generated second neuralnetwork model as the first neural network model 604 for training forothers (such as the second set of epochs 608B) of the plurality ofepochs, if the first set of impact scores 610A is less than the trainingimpact threshold as described, for example, in FIG. 6. Control may passto 1006 and the circuitry 202 may be configured to iterate theoperations from 1006.

At 1014, the trained second neural network model may be obtained basedon iterative control of the set of operations based on the comparison.In accordance with an embodiment, the circuitry 202 may be configured toobtain the trained second neural network model based on iterativecontrol of the set of operations (at 1006-1010) based on the comparison(i.e. the first set of impact scores 610A is more than the trainingimpact threshold performed at 1010). Control may pass to end.

Although the flowchart 1000 is illustrated as discrete operations, suchas 1004, 1006, 1008, 1010, 1012, and 1014, the disclosure is not solimited. Accordingly, in certain embodiments, such discrete operationsmay be further divided into additional operations, combined into feweroperations, or eliminated, depending on the particular implementationwithout detracting from the essence of the disclosed embodiments.

Various embodiments of the disclosure may provide a non-transitorycomputer-readable medium having stored thereon, computer-executableinstructions that when executed by an electronic device (e.g., theelectronic device 102), may cause the electronic device 102 to executeoperations. The operations may include storage of a first neural networkmodel (such as the first neural network model 104) trained for aclassification task of a real-time application. The first neural networkmodel 104 may be trained with a first plurality of training data points(such as the first plurality of training data points 106). Theoperations may further include retrieval of a first plurality ofexternal data points (such as the first plurality of external datapoints 110), which may be different from the first plurality of trainingdata points 106 on which the first neural network model 104 may betrained. The operations may further include application of the firstneural network model 104 on the first plurality of external data points110 to determine a first plurality of impact scores for each of thefirst plurality of external data points 110. The first plurality ofimpact scores may indicate a first amount of contribution of each of thefirst plurality of training data points 106 of the first neural networkmodel 104 towards a prediction of each of the first plurality ofexternal data points 110. The operations may further include selectionof a first set of external data points from the first plurality ofexternal data points 110 based on the determined first plurality ofimpact scores for each of the first plurality of external data points110. Further, the operations may include update of the first pluralityof training data points 106 with the selected first set of external datapoints to generate a second plurality of training data points. Theoperations may further include re-training of the first neural networkmodel 104 with the generated second plurality of training data points.

Various embodiments of the disclosure may provide a non-transitorycomputer-readable medium having stored thereon, computer-executableinstructions that when executed by an electronic device (e.g., theelectronic device 102), may cause the electronic device 102 to executeoperations. The operations may include storage of a first neural networkmodel (such as the first neural network model 604) trained for aclassification task of a real-time application. The operations mayfurther include selection of the first neural network model 604 fortraining for a plurality of epochs based on a first plurality oftraining data points (such as, the first plurality of training datapoints 606). The operations may further include a control of a set ofoperations for training the selected first neural network model 604. Theset of operations may include control of the training of the firstneural network model 604 for one or more epochs of the plurality ofepochs to generate a second neural network model trained on the firstplurality of training data points 606 for the one or more epochs. Theset of operations may further include determination of an impact scorefor each of the first plurality of training data points 606. The impactscore may indicate an amount of contribution of each of the firstplurality of training data points 606 of the generated second neuralnetwork model towards a prediction of each of the first plurality oftraining data points 606. The set of operations may further include are-selection of the generated second neural network model as the firstneural network model 604 for training for others of the plurality ofepochs, based on a comparison between the determined impact score and atraining impact threshold. The operations may further include obtainingthe trained second neural network model based on iterative control ofthe set of operations based on the comparison.

Exemplary aspects of the disclosure may include an electronic device(such as the electronic device 102). The electronic device 102 mayinclude a memory (such as the memory 204) configured to store a firstneural network model (such as the first neural network model 104) thatmay be trained for a classification task of a real-time application. Thefirst neural network model 104 may be trained with a first plurality oftraining data points (such as the first plurality of training datapoints 106). The electronic device 102 may further include circuitry(such as the circuitry 202) communicatively coupled to the memory 204.The circuitry 202 may be configured to retrieve a first plurality ofexternal data points (such as the first plurality of external datapoints 110) which may be different from the first plurality of trainingdata points 106 on which the first neural network model 104 may betrained. The circuitry 202 may be further configured to apply the firstneural network model 104 on the first plurality of external data points110 to determine a first plurality of impact scores for each of thefirst plurality of external data points 110. The first plurality ofimpact scores may indicate a first amount of contribution of each of thefirst plurality of training data points 106 of the first neural networkmodel 104 towards a prediction of each of the first plurality ofexternal data points 110. The circuitry 202 may be further configured toselect a first set of external data points from the first plurality ofexternal data points 110 based on the determined first plurality ofimpact scores for each of the first plurality of external data points110. The circuitry 202 may be further configured to update the firstplurality of training data points 106 with the selected first set ofexternal data points to generate a second plurality of training datapoints. The circuitry 202 may be further configured to re-train thefirst neural network model 104 with the generated second plurality oftraining data points.

In accordance with an embodiment, the circuitry 202 may be configured toapply a mathematical function to the determined first plurality ofimpact scores. The mathematical function comprises one of a maximafunction, an average function, a mean function, or a summation function.The circuitry 202 may be further configured to determine a first impactscore for each of the first plurality of external data points 110 basedon the application of the mathematical function.

In accordance with an embodiment, a first neural network model (such asthe first neural network model 404A) may be trained on a first pluralityof training data points (such as the first plurality of training datapoints 412). The circuitry 202 may be configured to apply the firstneural network model 404A on each of first plurality of external datapoints (such as first plurality of external data points 402A), togenerate a prediction score for each of the first plurality of externaldata points 402A. The circuitry 202 may be further configured to selectthe first set of external data points from the first plurality ofexternal data points 402A based on the determined first impact score andthe generated prediction score for each of the first plurality ofexternal data points 502A.

In accordance with an embodiment, the first amount of contribution mayindicate a number of features of each of the first plurality of trainingdata points 412 that may have contributed towards the prediction of eachof the first plurality of external data points 402A.

In accordance with an embodiment, the circuitry 202 may be configured toselect the first set of external data points from the first plurality ofexternal data points 502A based on a first range of impact scores. Theimpact score for each external data point in the selected first set ofexternal data points may lie within the first range of impact scores.

In accordance with an embodiment, the circuitry 202 may be configured toselect the first set of external data points from the first plurality ofexternal data points 402A based on a first impact score threshold. Theimpact score for each external data point in the selected first set ofexternal data points may lie above the first impact score threshold.

In accordance with an embodiment, the first plurality of training datapoints 412 and the first plurality of external data points 402Acorrespond to one of image data, audio data, text data, orthree-dimensional (3D) data. In accordance with an embodiment, thereal-time application comprises one of an image classification, a speechrecognition, or text recognition performed by the first neural networkmodel 104.

In accordance with an embodiment, the circuitry 202 may be furtherconfigured to apply a plurality of realistic variations to one or moreexternal data points of a first plurality of external data points (suchas, the first plurality of external data points 502A) to generate asecond plurality of external data points (such as, the second pluralityof external data points 516). The one or more external data points ofthe generated second plurality of external data points 516 maycorrespond to different realistic variation of the plurality ofrealistic variations. The circuitry 202 may be further configured toapply the first neural network model 404A on the generated secondplurality of external data points 516 to determine a second plurality ofimpact scores for each of the generated second plurality of externaldata points 516. The second plurality of impact scores may indicate asecond amount of contribution of each of the first plurality of trainingdata points 412 of the first neural network model 404A towards aprediction of each of the generated second plurality of external datapoints 516. The circuitry 202 may be further configured to select asecond set of external data points from the generated second pluralityof external data points 516 based on the determined second plurality ofimpact scores for each of the generated second plurality of externaldata points 516. The circuitry 202 may be further configured to updatethe first plurality of training data points 412 with the selected secondset of external data points to generate a third plurality of trainingdata points. The circuitry 202 may be further configured to re-train thefirst neural network model 404A with the generated third plurality oftraining data points.

In accordance with an embodiment, the plurality of realistic variationsinclude, but is not limited to, one of: a rotation variation, ahorizontal translation variation, a vertical translation variation, ashear variation, a zoom variation, a brightness variation, a contrastvariation, a flip variation, a sharpness variation, or a colorvariation. In accordance with an embodiment, a first variation of afirst external data point, of the selected second set of external datapoints, may be same as a second variation of a second external datapoint of the selected second set of external data points. The pluralityof realistic variations may include the first variation and the secondvariation.

In accordance with an embodiment, the circuitry 202 may be furtherconfigured to determine, based on the application of a first neuralnetwork model (such as the first neural network model 302B) on the firstplurality of training data points, a third plurality of impact scoresfor each training data point of the first plurality of training datapoints (such as, the first plurality of training data points 302A). Thethird plurality of impact scores may indicate a third amount ofcontribution of each of the first plurality of training data points 302Aof the first neural network model 302B towards prediction of each of thefirst plurality of training data points 302A. The circuitry 202 may befurther configured to select a set of training data points from thefirst plurality of training data points 302A based on the determinedthird plurality of impact scores for each of the first plurality oftraining data points 302A. The circuitry 202 may be further configuredto update the first plurality of training data points 302A with removalof the selected set of training data points to generate a fourthplurality of training data points. The circuitry 202 may be furtherconfigured to re-train the first neural network model 302B with thegenerated fourth plurality of training data points.

In accordance with an embodiment, the circuitry 202 may be furtherconfigured to select the set of training data points from the firstplurality of training data points 302A based on a second impact scorethreshold. The impact score for each training data point in the selectedset of training data points may lie below the second impact scorethreshold.

In accordance with an embodiment, the circuitry 202 may be furtherconfigured to generate a first graphical representation (such as thefirst graphical representation 302C) that may have a plurality of nodesand a plurality of edges between the plurality of nodes. Each node ofthe plurality of nodes may represent a training data point of the firstplurality of training data points 302A, and each edge of the pluralityof edges represents an impact score which indicates the third amount ofcontribution of each of the first plurality of training data points 302Aof the first neural network model 302B towards prediction of thecorresponding training data point.

In accordance with an embodiment, the circuitry 202 may be furtherconfigured to apply the re-trained first neural network model 104 on theselected first set of external data points to generate a set ofprediction scores to validate the prediction for the first set ofexternal data points.

In accordance with an embodiment, the circuitry 202 may be furtherconfigured to apply a plurality of second neural network models (suchas, the plurality of second neural network models 708A-708N) on a secondexternal data point (such as, the second external data point 710). Eachof the plurality of second neural network models 708A-708N may betrained on a plurality of training data points. The circuitry 202 may befurther configured to determine, based on the application of each of theplurality of second neural network models 708A-708N on the secondexternal data point 710, a plurality of impact scores for the secondexternal data point 710. The plurality of impact scores may indicate anamount of contribution of the plurality of training data points of eachof the plurality of second neural network models 708A-708N towards aprediction of the second external data point 710. The circuitry 202 maybe further configured to select a neural network model from theplurality of second neural network models 708A-708N based on theplurality of impact scores determined for the second external data point710 with respect to the plurality of second neural network models708A-708N. The circuitry 202 may be further configured to control thememory 204 to store the selected neural network model.

Exemplary aspects of the disclosure may include an electronic device(such as the electronic device 602). The electronic device 602 mayinclude a memory (such as the memory 204) configured to store a firstneural network model (such as the first neural network model 604)trained for a classification task of a real-time application. Theelectronic device 602 may further include circuitry (such as thecircuitry 202) communicatively coupled to the memory 204. The circuitry202 may be configured to select the first neural network model 604 fortraining for a plurality of epochs based on a first plurality oftraining data points (such as, the first plurality of training datapoints 606). The circuitry 202 may be configured to control a set ofoperations for training the selected first neural network model 604. Theset of operations may include control of the training of the firstneural network model 604 for one or more epochs of the plurality ofepochs to generate a second neural network model trained on the firstplurality of training data points 606 for the one or more epochs. Theset of operations may further include determination of an impact scorefor each of the first plurality of training data points 606. The impactscore may indicate an amount of contribution of each of the firstplurality of training data points 606 of the generated second neuralnetwork model towards a prediction of each of the first plurality oftraining data points 606. The set of operations may further include are-selection of the generated second neural network model as the firstneural network model 604 for training for others of the plurality ofepochs, based on a comparison between the determined impact score and atraining impact threshold. The circuitry 202 may be further configuredto obtain the trained second neural network model based on iterativecontrol of the set of operations based on the comparison.

The present disclosure may be realized in hardware, or a combination ofhardware and software. The present disclosure may be realized in acentralized fashion, in at least one computer system, or in adistributed fashion, where different elements may be spread acrossseveral interconnected computer systems. A computer system or otherapparatus adapted to carry out the methods described herein may besuited. A combination of hardware and software may be a general-purposecomputer system with a computer program that, when loaded and executed,may control the computer system such that it carries out the methodsdescribed herein. The present disclosure may be realized in hardwarethat comprises a portion of an integrated circuit that also performsother functions.

The present disclosure may also be embedded in a computer programproduct, which comprises all the features that enable the implementationof the methods described herein, and which when loaded in a computersystem is able to carry out these methods. Computer program, in thepresent context, means any expression, in any language, code ornotation, of a set of instructions intended to cause a system withinformation processing capability to perform a particular functioneither directly, or after either or both of the following: a) conversionto another language, code or notation; b) reproduction in a differentmaterial form.

While the present disclosure is described with reference to certainembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted withoutdeparture from the scope of the present disclosure. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the present disclosure without departure from itsscope. Therefore, it is intended that the present disclosure not belimited to the particular embodiment disclosed, but that the presentdisclosure will include all embodiments that fall within the scope ofthe appended claims.

What is claimed is:
 1. An electronic device, comprising: a memoryconfigured to store a first neural network model trained for aclassification task of a real-time application, wherein the first neuralnetwork model is trained with a first plurality of training data points;and circuitry communicatively coupled to the memory, wherein thecircuitry is configured to: retrieve a first plurality of external datapoints which are different from the first plurality of training datapoints on which the first neural network model is trained; apply thefirst neural network model on the first plurality of external datapoints to determine a first plurality of impact scores for each of thefirst plurality of external data points, wherein the first plurality ofimpact scores indicate a first amount of contribution of each of thefirst plurality of training data points of the first neural networkmodel towards prediction of each of the first plurality of external datapoints; select a first set of external data points from the firstplurality of external data points based on the determined firstplurality of impact scores for each of the first plurality of externaldata points; update the first plurality of training data points with theselected first set of external data points to generate a secondplurality of training data points; and re-train the first neural networkmodel with the generated second plurality of training data points. 2.The electronic device according to claim 1, wherein the circuitry isfurther configured to: apply a mathematical function to the determinedfirst plurality of impact scores, wherein the mathematical functioncomprises one of a maxima function, an average function, a meanfunction, or a summation function; and determine a first impact scorefor each of the first plurality of external data points based on theapplication of the mathematical function.
 3. The electronic deviceaccording to claim 2, wherein the circuitry is further configured to:apply the first neural network model, trained on the first plurality oftraining data points, on each of the first plurality of external datapoints, to generate a prediction score for each of the first pluralityof external data points; and select the first set of external datapoints from the first plurality of external data points based on thedetermined first impact score and the generated prediction score foreach of the first plurality of external data points.
 4. The electronicdevice according to claim 1, wherein the first amount of contributionindicates a number of features of each of the first plurality oftraining data points contributed towards the prediction of each of thefirst plurality of test data points.
 5. The electronic device accordingto claim 1, wherein the circuitry is further configured to select thefirst set of external data points from the first plurality of externaldata points based on a first range of impact scores, and wherein animpact score for each external data point in the selected first set ofexternal data points lies within the first range of impact scores. 6.The electronic device according to claim 1, wherein the circuitry isfurther configured to select the first set of external data points fromthe first plurality of external data points based on a first impactscore threshold, and wherein an impact score for each external datapoint in the selected first set of external data points lies above thefirst impact score threshold.
 7. The electronic device according toclaim 1, wherein the first plurality of training data points and thefirst plurality of external data points correspond to one of image data,audio data, text data, or three-dimensional (3D) data.
 8. The electronicdevice according to claim 1, wherein the real-time application comprisesone of an image classification, a speech recognition, or textrecognition performed by the first neural network model.
 9. Theelectronic device according to claim 1, wherein the circuitry is furtherconfigured to: apply a plurality of realistic variations to one or moreexternal data points of the first plurality of external data points togenerate a second plurality of external data points, wherein one or moreexternal data points of the generated second plurality of external datapoints correspond to different realistic variation of the plurality ofrealistic variations; apply the first neural network model on thegenerated second plurality of external data points to determine a secondplurality of impact scores for each of the generated second plurality ofexternal data points, wherein the second plurality of impact scoresindicate a second amount of contribution of each of the first pluralityof training data points of the first neural network model towardsprediction of each of the generated second plurality of external datapoints; select a second set of external data points from the generatedsecond plurality of external data points based on the determined secondplurality of impact scores for each of the generated second plurality ofexternal data points; update the first plurality of training data pointswith the selected second set of external data points to generate a thirdplurality of training data points; and re-train the first neural networkmodel with the generated third plurality of training data points. 10.The electronic device according to claim 9, wherein the first pluralityof training data points and the first plurality of external data pointscorrespond to one of image data or three-dimensional (3D) data, andwherein the plurality of realistic variations comprise one of a rotationvariation, a horizontal translation variation, a vertical translationvariation, a shear variation, a zoom variation, a brightness variation,a contrast variation, a flip variation, a sharpness variation, or acolor variation.
 11. The electronic device according to claim 9, whereinthe first plurality of training data points and the first plurality ofexternal data points correspond to audio data, and wherein the pluralityof realistic variations comprise one of a volume variation, a frequencyvariation, a tone variation, an audio equitization variation, or audionoise variation.
 12. The electronic device according to claim 9, whereina first variation of a first external data point, of the selected secondset of external data points, is same as a second variation of a secondexternal data point of the selected second set of external data points,and wherein the plurality of realistic variations include the firstvariation and the second variation.
 13. The electronic device accordingto claim 9, wherein a first variation of a first external data point, ofthe selected second set of external data points, is different from asecond variation of a second external data point of the selected secondset of external data points, and wherein the plurality of realisticvariations include the first variation and the second variation.
 14. Theelectronic device according to claim 1, wherein the circuitry is furtherconfigured to: determine, based on the application of the first neuralnetwork model on the first plurality of training data points, a thirdplurality of impact scores for each training data point of the firstplurality of training data points, wherein the third plurality of impactscores indicate a third amount of contribution of each of the firstplurality of training data points of the first neural network modeltowards prediction of each of the first plurality of training datapoints; select a set of training data points from the first plurality oftraining data points based on the determined third plurality of impactscores for each of the first plurality of training data points; updatethe first plurality of training data points with removal of the selectedset of training data points to generate a fourth plurality of trainingdata points; and re-train the first neural network model with thegenerated fourth plurality of training data points.
 15. The electronicdevice according to claim 14, wherein the circuitry is furtherconfigured to select the set of training data points from the firstplurality of training data points based on a second impact scorethreshold, and wherein an impact score for each training data point inthe selected set of training data points lies below the second impactscore threshold.
 16. The electronic device according to claim 14,wherein the circuitry is further configured to generate a firstgraphical representation having a plurality of nodes and a plurality ofedges between the plurality of nodes, and wherein each node of theplurality of nodes represents a training data point of the firstplurality of training data points, and each edge of the plurality ofedges represents an impact score which indicates the third amount ofcontribution of each of the first plurality of training data points ofthe first neural network model towards prediction of the correspondingtraining data point.
 17. The electronic device according to claim 1,wherein the circuitry is further configured to apply the re-trainedfirst neural network model on the selected first set of external datapoints to generate a set of prediction scores to validate the predictionfor the first set of external data points.
 18. The electronic deviceaccording to claim 1, wherein the circuitry is further configured to:apply a plurality of second neural network models on a second externaldata point, wherein each of the plurality of second neural networkmodels are trained on a plurality of training data points; determine,based on the application of each of the plurality of second neuralnetwork models on the second external data point, a plurality of impactscores for the second external data point, wherein the plurality ofimpact scores indicate an amount of contribution of the plurality oftraining data points of each of the plurality of second neural networkmodels towards prediction of the second external data point; select aneural network model from the plurality of second neural network modelsbased on the plurality of impact scores determined for the secondexternal data point with respect to the plurality of second neuralnetwork models; and control the memory to store the selected neuralnetwork model.
 19. An electronic device, comprising: a memory includinga first neural network model for a classification task of a real-timeapplication; and circuitry communicatively coupled to the memory,wherein the circuitry is configured to: select the first neural networkmodel for training for a plurality of epochs based on a first pluralityof training data points; control a set of operations for training theselected first neural network model, wherein the set of operationscomprise: control the training of the first neural network model for oneor more epochs of the plurality of epochs to generate a second neuralnetwork model trained on the first plurality of training data points forthe one or more epochs, determine an impact score for each of the firstplurality of training data points, wherein the impact score indicates anamount of contribution of each of the first plurality of training datapoints of the generated second neural network model towards predictionof each of the first plurality of training data points, and re-selectthe generated second neural network model as the first neural networkmodel for training for others of the plurality of epochs, based on acomparison between the determined impact score and a training impactthreshold; and obtain the trained second neural network model based oniterative control of the set of operations based on the comparison. 20.A method, comprising: in an electronic device: storing a first neuralnetwork model trained for a classification task of a real-timeapplication, wherein the first neural network model is trained with afirst plurality of training data points; and retrieving a firstplurality of external data points which are different from the firstplurality of training data points on which the first neural networkmodel is trained; applying the first neural network model on the firstplurality of external data points to determine a first plurality ofimpact scores for each of the first plurality of external data points,wherein the first plurality of impact scores indicate a first amount ofcontribution of each of the first plurality of training data points ofthe first neural network model towards prediction of each of the firstplurality of external data points; selecting a first set of externaldata points from the first plurality of external data points based onthe determined first plurality of impact scores for each of the firstplurality of external data points; updating the first plurality oftraining data points with the selected first set of external data pointsto generate a second plurality of training data points; and re-trainingthe first neural network model with the generated second plurality oftraining data points.
 21. A non-transitory computer-readable mediumhaving stored thereon computer-executable instructions that, whenexecuted by an electronic device, causes the electronic device toexecute operations, the operations comprising: storing a first neuralnetwork model trained for a classification task of a real-timeapplication, wherein the first neural network model is trained with afirst plurality of training data points; and retrieving a firstplurality of external data points which are different from the firstplurality of training data points on which the first neural networkmodel is trained; applying the first neural network model on the firstplurality of external data points to determine a first plurality ofimpact scores for each of the first plurality of external data points,wherein the first plurality of impact scores indicate a first amount ofcontribution of each of the first plurality of training data points ofthe first neural network model towards prediction of each of the firstplurality of external data points; selecting a first set of externaldata points from the first plurality of external data points based onthe determined first plurality of impact scores for each of the firstplurality of external data points; updating the first plurality oftraining data points with the selected first set of external data pointsto generate a second plurality of training data points; and re-trainingthe first neural network model with the generated second plurality oftraining data points.